How to use C# 4.0 tasks feature with these properties - assigning variables on run time - keep certain number of tasks alive

Alright i have been searching for days but there is no example of what i am trying to achieve. Currently i am able to use tasks for multi-threaded crawling but it is very bad written. You can see the whole code from here :

http://social.msdn.microsoft.com/Forums/en-US/csharpgeneral/thread/6fa6fd12-85c5-4489-81b1-25bc0126d406

Now coming my question. I want to use System.Threading.Ta开发者_开发百科sks.Task for multi-threading. Development is microsoft visual studio 2010 and C# 4.0.

What i need is

1-) I want to keep certain number of tasks running for all the time. So when a thread finished its job another thread should start working immediately. With this way always certain number of threads will be actively running.

2-) I need to be able to give tasks variables on run time. So think as i have a links pool. I started with first 100 but lets say 45 th task finished first. So another task will be created and this task will take number 101 th link. When another one finished it will continue as this way and always certain number of tasks will be alive.

3-) I need to be able to collect results of tasks. After a task finished somehow collect the result. This part seems like the easiest part.

Whichever task example i found just shows 2 task or 3 task , getting certain variables not changing on run time and not keeping certain numbers alive. Multi-threaded applications are the future but there is so bad documentation.

Have you looked into the TPL-DataFlow CPL yet:

TPL Dataflow (TDF) is a new .NET library for building concurrent applications. It promotes actor/agent-oriented designs through primitives for in-process message passing, dataflow, and pipelining. TDF builds upon the APIs and scheduling infrastructure provided by the Task Parallel Library (TPL) in .NET 4, and integrates with the language support for asynchrony provided by C#, Visual Basic, and F#.

It's beta but I think your problem are not the threads. Seems you are trying to build some kind of Agent-based programming model and this DataFlow gives you some really nice tools for this.

Here is the Homepage and here is a nice video on this.

Why do you NEED a certain number of tasks? Let the pool/API decide.

Passing data to tasks is just like passing data to threads: share a data structure.

Use Task to return a result from a task

EDIT

You cannot set the number of worker threads or the number of I/O completion threads to a number smaller than the number of processors in the computer.

If the common language runtime is hosted, for example by Internet Information Services (IIS) or SQL Server, the host can limit or prevent changes to the thread pool size.

Use caution when changing the maximum number of threads in the thread pool. While your code might benefit, the changes might have an adverse effect on code libraries you use.

Setting the thread pool size too large can cause performance problems. If too many threads are executing at the same time, the task switching overhead becomes a significant factor.

(from the MSDN)

Really, you can never force a specific number of tasks to be active. Just create the tasks and allow the run-time to decide which ones should/can run.

If you don't like that, do not use the TPL and bring your own threads.