Help with deciphering WCF timeout errors

I have a windows service that is very heavily multithreaded (hundreds to thousands at the same time). Those threads scan different machines and call WCF web services of one Web Server. Lately I started getting Timeout errors. The thing that confuses me is

The request channel timed out while waiting for a reply after 00:02:41.8806080. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a po开发者_JS百科rtion of a longer timeout. Server stack trace: .... Exception rethrown at [0]: ... The HTTP request to 'http://XXX/XXX/MonitorService.svc' has exceeded the allotted timeout of 00:10:00. The time allotted to this operation may have been a portion of a longer timeout. at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)

My question is what happened during over 7 minutes? Was it waiting at the client for 10 minutes but at the web server for 2:41? Why would that be? If I call a webservice hundreds of times from different thread at the same time is it somehow queued on the client? Even before reaching IIS?

I appreciate any help with this. Thanks so much in advance.

This is very hard to diagnose. There can be plenty of reasons why this happens and you have to find better information about your system before you will be able to clearly say what is going on. You should use performance counters to monitor number of web service calls and processing on web server. The reasons for timeouts can be:

• Throttling on service which can process only limited number of requests or hold only limited number of instances and sessions concurrently. Other requests are waiting in queue on server. Together with slow processing this can be big limitation.
• Concurrency settings for singleton services
• Network bandwith if your windows service floods the network with many big messages
• HTTP persistant connections - by default machine can create only two parallel connections to the same server over HTTP. Other request are waiting in queue on client.
• etc. + combinations

I got a WCF timeout error with no other exceptions being thrown. It ends up that the tags I added to the config file to enable tracing somehow limit the payload size and just prevent a response form being sent. As soon as I commented out the tags, the normally expected response resumed.