WebClient.DownloadFile in BackgroundWorker performance unacceptable
I have a WinForms application that needs to download many (tens of thousands) fairly large (multi-MB) files per day. I wrote a simple test using the following code:
using (var wc = new System.Net.WebClient())
{
foreach (string url in UrlsToDownload())
{
string targetPath = SafeFilePathOf(url);
wc.DownloadFile(url, targetPath);
}
}
The target machine has a 1 Gb/s connection, but testing shows a sustained download of about 1MB/s. This is less than what I was expecting, but the source 开发者_运维问答servers may have slower connections. A full day's download will require several hours of connectivity, which is acceptable. Network utilization is fairly constant at about 1%:
However, I need to perform downloads in a background thread and support cancelling and download progress. The .NetSystem.ComponentModel.BackgroundWorker
seems designed for this, so I put the exact same code in a BackgroundWorker
instance and call RunWorkerAsync
:
Download performance plummets to about 0.05 MB/s. A day's work will require about a week to perform; this is not going to fly.
Why is BackgroundWorker
performance so bad? Neither the CPU nor network is overloaded. The application is not blocked, I simply moved the download code from the UI thread to a BackgroundWorker
. Calling backgroundWorker.Priority = ThreadPriority.AboveNormal
has no effect.
Increase the connection limit on the WebClient:
internal class CustomWebClient : WebClient
{
public CustomWebClient()
{
}
protected override WebRequest GetWebRequest(Uri uri)
{
HttpWebRequest w = (HttpWebRequest)base.GetWebRequest(uri);
w.ServicePoint.ConnectionLimit = 100;
return w;
}
}
Network activity performs same whether it's background thread or UI thread. The problem can be any other code of reporting progress or logging activity etc. That may block the thread.
精彩评论