System.OutOfMemoryException with WCF during request

I have a server that is throwing an System.OutOfMemoryException, from what i can tell, during the transmission of data during a request. From looking at this trace, is my assumption right?

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
  at System.Net.Security.NegoState.ProcessAuthentication(LazyAsyncResult lazyResult)
  at System.Net.Security.NegotiateSt开发者_StackOverflow社区ream.AuthenticateAsServer(NetworkCredential credential, ProtectionLevel requiredProtectionLevel, TokenImpersonationLevel requiredImpersonationLevel)
  at System.ServiceModel.Channels.WindowsStreamSecurityUpgradeProvider.WindowsStreamSecurityUpgradeAcceptor.OnAcceptUpgrade(Stream stream, SecurityMessageProperty& remoteSecurity)
  at System.ServiceModel.Channels.StreamSecurityUpgradeAcceptorBase.AcceptUpgrade(Stream stream)
  at System.ServiceModel.Channels.InitialServerConnectionReader.UpgradeConnection(IConnection connection, StreamUpgradeAcceptor upgradeAcceptor, IDefaultCommunicationTimeouts defaultTimeouts)
  at System.ServiceModel.Channels.ServerSessionPreambleConnectionReader.ServerFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
  at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
  at System.ServiceModel.Channels.CommunicationObject.Open()
  at System.ServiceModel.Dispatcher.ChannelHandler.OpenAndEnsurePump()
  at System.ServiceModel.Dispatcher.ChannelHandler.OpenAndEnsurePump(Object state)
  at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
  at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.OnSecurityContextCallback(Object o)
  at System.Security.SecurityContext.Run(SecurityContext securityContext, ContextCallback callback, Object state)
  at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
  at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
  at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object state)
  at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
  at System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
  at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

I have maxArrayLength, and maxStringContentLength set really high in my config file. Is it possible that one of these is extremely huge, and causing the channel to throw this error? This is a .NET WCF service that is communicating locally through IPC with another process.

Given that this is a production issue and its happening in different places sporadically I'm guessing you're having a high memory utilization issue. I would obtain a Memory dump of the system and use WinDbg to see what's causing this.

You can use proc dump or Debug Diag to do this (but make sure you use the 32 or 64 bit one depending on which one you need).

If you've never done analysis on memory before (and its really not easy) then you might want to take a look at this article