C# How to loop a large set of folders and files recursively without using a huge amount of memory

I want to index all my music files and store them in a database. I have this function that i call recusively, starting from the root of my music drive.

i.e.

start > ReadFiles(C:\music\);

ReadFiles(path){
   foreach(file)
      save to index;

   foreach(directory)
      ReadFiles(directory);
}

This works fine, but while running the program the amount of memory that is used g开发者_C百科rows and grows and.. finally my system runs out of memory.

Does anyone have a better approach that doesnt need 4GB of RAM to complete this task?

Best Regards, Tys

Alxandr's queue based solution should work fine.

If you're using .NET 4.0, you could also take advantage of the new Directory.EnumerateFiles method, which enumerates files lazily, without loading them all in memory:

void ReadFiles(string path)
{
    IEnumerable<string> files =
        Directory.EnumerateFiles(
            path,
            "*",
            SearchOption.AllDirectories); // search recursively

    foreach(string file in files)
        SaveToIndex(file);
}

Did you check for the . and .. entries that show up in every directory except the root?

If you don't skip those, you'll have an infinite loop.

You can implement this as a queue. I think (but I'm not sure) that this will save memory. At least it will free up your stack. Whenever you find a folder you add it to the queue, and whenever you find a file you just read it. This prevents recursion.

Something like this:

Queue<string> dirs = new Queue<string>();
dirs.Enqueue("basedir");
while(dirs.Count > 0) {
    foreach(directory)
        dirs.Enqueue(directory);
    ReadFiles();
}

Beware, though, that EnumerateFiles() will stop running if you don't have access to a file or if a path is too long or if some other exception occurs. This is what I use for the moment to solve those problems:

public static List<string> getFiles(string path, List<string> files)
{
    IEnumerable<string> fileInfo = null;
    IEnumerable<string> folderInfo = null;
    try
    {
        fileInfo = Directory.EnumerateFiles(str);
    }
    catch
    {

    }
    if (fileInfo != null)
    {
        files.AddRange(fileInfo);
        //recurse through the subfolders
        fileInfo = Directory.EnumerateDirectories(str);
        foreach (string s in folderInfo)
        {
            try
            {
                getFiles(s, files);
            }
            catch
            {

            }
        }
    }
    return files;
}

Example use:

List<string> files = new List<string>();
files = folder.getFiles(path, files);

My solution is based on the code at this page: http://msdn.microsoft.com/en-us/library/vstudio/bb513869.aspx.

Update: A MUCH faster method to get files recursively can be found at http://social.msdn.microsoft.com/Forums/vstudio/en-US/ae61e5a6-97f9-4eaa-9f1a-856541c6dcce/directorygetfiles-gives-me-access-denied?forum=csharpgeneral. Using Stack is new to me (I didn't even know it existed), but the method seems to work. At least it listed all files on my C and D partition with no errors.

It could be junction folders wich leads to infinite loop when doing recursion but i am not sure , check this out and see by yourself . Link: https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/mklink