Timeout problem with ASP.NET web scraping

I'm trying to implement a web scraping crawler as a part of my ASP.NET MVC project. It works with large data gathered from different URLS using Html Agility Pack. The problem is when I want to actually run the function I get "The connection was reset" from remote server after a minute. I'm getting better result when I run it locally. I开发者_JS百科 have access to remote IIS. Any suggestion to solve this problem and/or any alternatives?

If you have a long running process in ASP.NET, it is best to let it run on a different thread.

See this and this - related questions and this MSDN article.

Connection and network problems could result in such problems. To avoid blocking scraping of other urls you could parallelize the work into separate threads.