ThreadPool with multiple threads creating FTP requests times out

I'm trying to create a collection of FTP web requests to download a collection of files.

Was working correctly doing this in a single thread but am trying to do with multiple threads now but am getting a timeout exception. I think I'm missing something pretty simple but cannot seem to work it out

Here is code:

internal static void DownloadLogFiles(IEnumerable<string> ftpFileNames, string localLogsFolder)
{
    BotFinder.DeleteAllFilesFromDirectory(localLogsFolder);

    var ftpWebRequests = new Collection<FtpWebRequest>();

    // Create web request for each log filename
    foreach (var ftpWebRequest in ftpFileNames.Select(filename => (FtpWebRequest) WebRequest.Create(filename)))
    {
        ftpWebRequest.Credentials = new NetworkCredential(BotFinderSettings.FtpUserId, BotFinderSettings.FtpPassword);
        ftpWebRequest.KeepAlive = false;
        ftpWebRequest.UseBinary = true;
        ftpWebRequest.CachePolicy = NoCachePolicy;
        ftpWebRequest.Method = WebRequestMethods.Ftp.DownloadFile;
        ftpWebRequests.Add(ftpWebRequest);
    }

    var threadDoneEvents = new ManualResetEvent[ftpWebRequests.Count];

    for (var x = 0; x < ftpWebRequests.Count; x++)
    {
        var ftpWebRequest = ftpWebRequests[x];
        threadDoneEvents[x] = new ManualResetEvent(false);
        var threadedFtpDownloader = new ThreadedFtpDownloader(ftpWebRequest, threadDoneEvents[x]);
        ThreadPool.QueueUserWorkItem(threadedFtpDownloader.PerformFtpRequest, localLogsFolder);               
    }

    WaitHandle.WaitAll(threadDoneEvents);
}

class ThreadedFtpDownloader
{
    private ManualResetEvent threadDoneEvent;
    private readonly FtpWebRequest ftpWebRequest;

    /// <summary>
    /// 
    /// </summary>
    public ThreadedFtpDownloader(FtpWebRequest ftpWebRequest, ManualResetEvent threadDoneEvent)
    {
        this.threadDoneEvent = threadDoneEvent;
        this.ftpWebRequest = ftpWebRequest;
    }

    /// <summary>
    /// 
    /// </summary>
    /// <param name="localLogsFolder">
    /// 
    /// </param>
    internal void PerformFtpRequest(object localLogsFolder)
    {
        try
        {
            // TIMEOUT IS HAPPENING ON LINE BELOW
            using (var response = ftpWebRequest.GetResponse())
            {
                using (var responseStream = response.GetResponseStream())
                {
                    const int length = 1024*10;
                    var buffer = new Byte[length];
                    var bytesRead = responseStream.Read(buffer, 0, length);

                    var logFileToCreate = string.Format("{0}{1}{2}", localLogsFolder,
                                        ftpWebRequest.RequestUri.Segments[3].Replace("/", "-"),
                                        ftpWebRequest.RequestUri.Segments[4]);

                    using (var writeStream = new FileStream(logFileToCreate, FileMode.OpenOrCreate))
                    {
                        while (bytesRead > 0)
                        {
                            writeStream.Write(buffer, 0, bytesRead);
                            bytesRead 开发者_StackOverflow中文版= responseStream.Read(buffer, 0, length);
                        }
                    }
                }
            }

            threadDoneEvent.Set();
        }
        catch (Exception exception)
        {
            BotFinder.HandleExceptionAndExit(exception);
        }
    }
}

It seems to be downloading the first two files (using two threads I'm assuming) but then timeout seems to occur when these complete and application tries to move onto next file.

I can confirm that the FTPWebRequest which is timing out is valid and the file exists, I think I may have an open connection or something.

Was going to post a comment but probably easier to read in an answer:

Firstly, if I set the ftpRequest.Timout property to Timeout.Infinite, the timeout issue disappears however having an infinite timeout is probably not best practice. So I'd prefer to go about solving this another way...

Debugging the code, I can see that when it gets to:

ThreadPool.QueueUserWorkItem(threadedFtpDownloader.PerformFtpRequest, localLogsFolder);

It enters into PerformFtpRequest method for each FTP web request and calls the ftpWebRequest.GetResponse() but then only progresses further for the first two requests. The rest of the requests stay active but don't go any further until the first two finish. So this basically means they are left open while waiting for other requests to complete before starting.

I think the solution to this problem would either be allowing all the requests to execute at once (ConnectionLimit property is having no effect here) or to prevent execution from calling GetResponse until it's actually ready to use the response.

Any good ideas on best way to solve this? At the moment all I can seem to think of are hacky solutions which I'd like to avoid :)

Thanks!

You should get the ServicePoint for the request and set the ConnectionLimit

ServicePoint sp = ftpRequest.ServicePoint;
sp.ConnectionLimit = 10;

The default ConnectionLimit is 2 -- that's why you're seeing that behavior.

UPDATE: See this answer for a more thorough explanation:

How to improve the Performance of FtpWebRequest?