Multi-Threaded Component Design Question

We have 10 denormalized tables that contains large amounts of data (11,000,000) in many of them. The client wants the user to be able to download up to 100K of records 开发者_运维百科at a time. What I am wondering is the design of a background process to handle this. I was thinking of possibly setting up a queue for each of the tables and having multiple threads handling the requests. I'm definetly not an expert on designing multi-threaded components. Do you guys think I'm on the right track here? Any suggestions on implementation?

Thanks!

Design scenario

1- Since this is an Asp.Net applicaion , how many concurrent users will download the data?

2- Are you thinking to give the download data facility with in your application , did you thought about process re-cycling?

3- What if the user session expires but download is still in progress?

4- You probably moving towards a streaming server.

i think , you should think about these scenarion first than decide which type of application is actually required.

in my thoughts

i think , you should think of creating a windows based solution which will be downloaded to the client computer and made use of client resources (threads etc) to connect to the server and your server must support streaming.

Will the users be making custom queries or just downloading "chunks" of the table?

If the latter then why not just generate the CSV for each chunk ahead-of-time? You'll only need 110 chunks if each chunk has 100,000 records and the table only has 11,000,000 records in total.

I think this is one way of doing it.

You can call another page and put the process in next page.

Following article might be useful.

http://codebetter.com/blogs/peter.van.ooijen/archive/2006/06/20/146582.aspx