IIS 7 worker process bottleneck, large number of awaiting requests under application pool ASP.NET 3.5 + 2.0

I am using ASP.NET 2.0, .NET 2.0 Framework, and IIS 7. I am seeing a large queue of "requests" appear under the "worker process" option. State recorded appear to be Authenticate Request and Execute Request Handles more than anything else.

I have amended aspnet.config in C:\Windows\Microsoft.NET\Framework64\v2.0.50727 (32 bit path and 64 bit path) to include:

maxConcurrentRequestsPerCPU="50000"
maxConcurrentThreadsPerCPU="0"
requestQueueLimit="50000"

I have amended machine.config in C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG (32-bit and 64-bit path) to include:

autoConfig="false"
maxIoThreads="100"
maxWorkerThreads="100"
minIoThreads="50"
minWorkerThreads="50"
minFreeThreads="176"
minLocalRequestFreeThreads="152"

Still I get this issue.

The issue manifests itself as a large number of requests in the Worker Process queue.

The number of current connections to the website display 500 when this issue occurs. I don't think I have seen concurrent connections over 500 without this issue occurring.

The web application slows as the requests block.

Refreshing th开发者_JAVA百科e application pool resolves for a while (as expected) as the load is spread between the two pools.

The application pool in question FIXED REQUEST have been set to refresh on 50000.

Note: .NET 3.5 framework uses 2.0 framework appnet and machine configuration files, I believe.

Server resources (CPU, RAM) are not used to the full potential.

Can I suggest you take a look at the materials on Tess Ferrandez's site If It Is Broken, Fix It You Should.

What you want to be doing is capturing a dump of the worker process using ADPlus at the time you experience a high number of requests in the request queue and when the app starts to grind to a halt.

Once you grabbed this dump you want to load this into WinDBG+SOS and start tracking down the culprit.

Tess has a great series of labs about how to utilise these tools to great effect:

.NET Debugging Demos - Information and setup instructions

If you are able to, then you could also attach a profiler to the application (such as RedGate's Performance Profiler to try and discover the root cause.

Follow the KB article http://support.microsoft.com/kb/821268 and update me if you are still facing the issue.

You can also try FREB tracing in IIS 7. http://learn.iis.net/page.aspx/266/troubleshooting-failed-requests-using-tracing-in-iis-7/

Ended up increasing the worker processes from 1 to 2 (web garden). Issue has not reoccurred since although were are using sessions but over the course of a month there have been no reports of session problems from end users.

EDITED TO ADD:

The specific issue was related to the generation of a extremely large CSV reports which are processed on the worker process server side, not a bug , just the way it works. HTML reports are fine, the XML transform is passed client side.

The separation of worker processes resolved the issues until the sever side xml transform to csv file creation can be passed to an addition process therefore removing any influence from other users. Was simply down to the size of the files / number of rows we were working with when trying to create CSV reports.