asp.net website stops responding when doing heavy query

I have an asp.net website that is nearing release. The site deals with business management. One of the things I am doing to prepare for release is upload a whole bunch of data from purchased lists of businesses into our SQL database. To do this, I'm using a very simple meth开发者_C百科od: I put the .csv files in a folder on the server, and use the Page_Load event on a temporary page to do the logic needed to process the data and put it in our database.

This is a lengthy process. the business lists are broken up into a whole bunch of files, and I've got my script on the page set up to import all the data from one file on each refresh of the page. It takes about 10 minutes per file.

The weird thing is this: When processing this data ( while the server is processing everything, before the page_load method terminates) the website is blocked - no other users can access it, and no other pages on the site will load. They get stuck on the "Connecting..." phase.

This isn't a huge problem right now, as the site is just running in a test environment. However, I may need to use similar scripts to put data into the database when the site is live, and obviously its not acceptable to have access to all users blocked when this is happening.

I am fairly sure this isn't a problem with the SQL Server, as I am able to do simultaneous queries while this process is running, so it must be something wrong with the web server.

I can't post a code exerpt, because it is long winded and technical. However, nothing difficult is going on: The method reads in the data from the .csv file, processes it, and makes an HTTP Request to an external service for each line of the file, to collect some more relevant data. Then, all the data is inserted into the SQL database.

As I understand it, web servers shouldn't block like this. They should be able to simulatenously serve pages to lots of users at once, even if one thread/user is requiring lots of resources.

Can any suggest what might be up? Where should I start looking for a solution?

Have the request be processed by an asynchrounous handler.

http://msdn.microsoft.com/en-us/library/ms227433%28v=vs.85%29.aspx

This will allow your website to continue doing it's business while the long process is running.

ASP.NET is not meant for heavy processing. The ASP.NET page should just trigger the action. A background process should do the heavy lifting. Both can communicate using a queue or something similar. The Web page should simply report the status of the background tasks.

This paradigm is central in modern computing (especially cloud computing).

Please refer to http://msdn.microsoft.com/en-us/library/ff647787.aspx#scalenetchapt06_topic8 from msdn.

I think the problem is that you are using the default settings for asp.net where you don't get enough threads and only 2 outbound tcp/ip connections. This is just horrible for a real website.

Please change maxconnections, maxworkerthreads and others in your machine.config. There is a formula for some of these settings like 12 * number of CPUs, etc.

This should give you the performance boost you need.

Hope this helps.

Edit:-

My comments were before I learnt about the autoconfig=true setting in the machine.config. I did some preliminary research and discovered that these settings work for most applications. so, these customized settings are only needed for specialized scenarios that might need carefully considered optimizations.