Upload files to Azure BLOB storage - Parallel.Foreach is slower than Foreach
I have the following code for uploading a folder form local storage to blob storage, Including the folder name itself in the blob's name ( The code is based on some methods found here http://blog.smarx.com/posts/pivot-odata-and-windows-azure-visual-netflix-browsing ) :
public static void UploadBlobDir(CloudBlobContainer container, string dirPath)
{
string dirName = new Uri(dirPath).Segments.Last();
Parallel.ForEach(enumerateDirectoryRecursive(dirPath), file =>
{
string blobName = Path.Combine(dirName, Path.GetFullPath(file)).Substring(dirPath.开发者_如何学运维Length - dirName.Length);
container.GetBlobReference(blobName).UploadFile(file);
});
}
and :
private static IEnumerable<string> enumerateDirectoryRecursive(string root)
{
foreach (var file in Directory.GetFiles(root))
yield return file;
foreach (var subdir in Directory.GetDirectories(root))
foreach (var file in enumerateDirectoryRecursive(subdir))
yield return file;
}
This code works and uploads the folder as intended but it takes an awful lot of time to complete - It takes 20 seconds to upload 25 files, 40KB~ each. So I tired to replace the parallel loop with a regular one like so :
foreach (var file in enumerateDirectoryRecursive(i_DirPath))
{
string blobName = Path.Combine(dirName, Path.GetFullPath(file)).Substring(i_DirPath.Length - dirName.Length);
container.GetBlobReference(blobName).UploadFile(file);
}
Now the uploading completes instantly (3 Seconds Approx.).
It's also important to note that I am working against the storage emulator for development.
The Parallel.Forech should obviously be faster. Does this difference comes form the storage emulator limitations (And when going live, Parallel would be faster) or is it something else I might be doing wrong ?In my experience, the storage emulator tells you strictly nothing about the performance you should expect (or not) from the actual Azure Storage. The emulator is typically extremely slow.
Then Parallel.Foreach
will only be faster if your transfer happen to be latency-bound rather than I/O bound. Then, please note that Parallel.Foreach
will only use your number of CPU as the default parallelization degree. For latency-bound processes, you should typically have much more threads than that, typically 4 to 8 threads per CPU (YMMV).
精彩评论