What is causing the bottleneck in this scenario?

I'm looking for some help with a production issue we have encountered.

We have an NServiceBus Handler that when it processes a message sends a request to an asmx (.net 2.0) webservice running on IIS 6.0, the web service then sends a second http request to a 3rd Party web service, on average the 3rd party service takes 500ms to respond but we have been told will only start queing requests when 11 simultaneous requests are in progress.

To meet our processing SLA's we need to be able to process 79,000 messages an hour or ~ 1300 messages a minute. With 11 requests being processed in parallel and taking an average of 500 ms this should be achievable if we can get eleven concurrent requests firing, every second we should be able to process about 20 messages.

The problem we have is that we cannot get anywhere near these figures. Our deployed solution is as follows. We have a service that开发者_JS百科 polls a database every 5 seconds to retrieve a batch of 100 new messages, it sends them to a distributor which load balances accross four worker windows services running on two servers. I'll call them NsbServerA & NSbServerB. Each worker service is configured with 2 threads so we have 8 threads in total. We have two servers runing the asmx web services I'll call the asmx1 & asmx2, ServerA sends requests to asmx1 and ServerB sends requests to asmx2.

The current throughput is 30,000 messages per hour, ~500 per min, ~8 per second so somewhere we have a bottle neck the question is where?

I have seen this article http://support.microsoft.com/default.aspx?scid=kb;en-us;821268 which seems to indicate that you have to tune the IIS 6.0 process model if you want to make more than a few of concurrent calls to the same webservice.

Quote from the article: "If you are making one Web service call to a single IP address from each ASPX page, Microsoft recommends that you use the following configuration settings: Set the values of the maxWorkerThreads parameter and the maxIoThreads parameter to 100. Set the value of the maxconnection parameter to 12*N (where N is the number of CPUs that you have). Set the values of the minFreeThreads parameter to 88*N and the minLocalRequestFreeThreads parameter to76*N. Set the value of minWorkerThreads to 50. Remember, minWorkerThreads is not in the configuration file by default. You must add it."

If anyone can shed some light on what is happening or what the solution is I would be really grateful.

Thanks,

Charlie

One likely possibility:

The ServicePointManager, through which all your .NET Web requests are routed, has a default of two concurrent connections per object. See ServicePointManager.DefaultConnectionLimit.

So if your 3rd party Web service takes an average of 500 ms to respond, a single server can only do four requests to that service every second. (i.e. two concurrent requests every 500 ms).

Multiply that by two servers and you get your ~8 messages per second.

Try setting ServicePointManager.DefaultConnectionLimit = 4; when you start your program, and see if that increases your throughput.

I would suggest testing each layer of the system in isolation.

Specifically, first discount the database tier as an issue. I'd put together some simple integration-style unit tests that exercise the peak throughput of the database, ideally by using something like Parallel.ForEach to run a few hundred threads concurrently.

Then move up a layer and use something like a stub to "mock out" the database calls and verify that the service tier is behaving as it should.

Then at least you'll be able to narrow down the inefficiencies and deal with them.

Sounds like it could be many things.

1. Are you using the commercially available Standard Edition of NServiceBus? Without that your handler will work single threaded.
2. Is your mid-tier web service asynchronous? If the second service round trip is made synchronously then a thread will be tied up and unable to process incoming requests.
3. Have you configured sufficient worker threads to process incoming requests? UPDATE: See this page http://msdn.microsoft.com/en-us/library/ff647787.aspx#scalenetchapt06_topic9 ("Table 6.1: Recommended Threading Settings for Reducing Contention") for high throughput recommendations. The settings you should look at are minFreeThreads, maxconnection, maxWorkerThreads, minWorkerThreads.

I would be nervous committing to a throughput SLA of 20 messages per second if you cannot reliably achieve this without massive optimization.

Also probably a stupid question but why can't you call the remote service from your message handler directly?