Threaded HTTP Post Application

I have an application that performs HTTP posts at regular intervals (data retrieved from SQL). Every 30 seconds a maximum of 50 threads are spawned and run HTTP posts concurrently. If a post fails, it waits 2x as long as the interval is set to. This will happen twice. So for instance, 30s, 60s then 120s.

I am using a normal Thread.Start() to get this process rolling, however I have found that on the live server, it completely annihilates the CPU.

My questions are as follows:

• Is there a better class to use for thread performance?
• Is there a way to limit a threaded applications CPU usage in .NET?

Thanks,

Kyle

You should not be using threads for running multiple I/O streams. Since those threads are mostly blocking on I/O, you can more efficiently do this use non-blocking or asynchronous I/O. Instead of having one thread talk with one server, you have one thread talk with N servers.

Since you are using HttpWebRequest, you want to look at HttpWebRequest.BeginGetResponse and HttpWebRequest.BeginGetRequestStream.

In general, Use the ThreadPool.

Inside every .NET process there's a thread pool, a pool of threads ready to work for you. You should use it.

For reasons why, see Thread vs ThreadPool , or The .NET ThreadPool

The short story is: it's much more efficient, and it's much kinder to your CPU, than your current approach of starting threads on demand.

.NET also has a evry easy-to-use mechanism for sending work to the threads in the threadpool: ThreadPool.QueueUserWorkItem.

I don't know how the .NET threadpool is managed internally, but I know that engineers smarter than me have done the analysis work to figure out what it should do, how many threads should be in it, how it should behave so as not to swamp the CPU. Yes, it's designed specifically to avoid the problem you described. If the threadpool is good enough for ASPNET worker processes, it's probably good enough for my apps.

And yours, too.

I am sorry, I don't have any experience with multi-threaded HTTP posts.

Having said that, which class(es) do you use for HTTP post?
I suppose, it should have methods to do HTTP post in asynchronous manner.

And, if that doesn't work, you could use ThreadPool instead of creating your own threads.

is it really necessary to have that 50 threads run simultaneously?

Just try to limit the number of spawned threads like just 10 threads and then run it by batch.

I'm wondering what else you might have running that clobbers CPU performance (such as an inefficient monitoring loop), or whether it's a page file thrashing issue because you've used up memory. If your worker class has a lot of local data you can soak up memory rather quickly.

I've done something similar using HttpWebRequest and have been able to have about 100 simultaneous connections in flight but at that point I've maxed out network bandwidth and memory, although 50 connections ran fine. I create my own threads instead of using the ThreadPool as I use a callback so I can easily track thread status and even abort threads if needed. It also simplifies restarting from a failed connection - I just throw the thread back into a queue of pending threads.

The ThreadPool is good for preventing you from launching too many threads, but I think the thread creation overhead is insignificant compared to the network connection response delay, at least in my experience.