Reducing the time needed to gather files from a larger amount of directories

Question

Hiho.

I'm not sure if it is possible to decrease loading times in my program, or if I just have to switch to SSDs. But maybe someone has an Idea :)

I'm getting the first (max) 100 files from a large number of folders and write their path to a single list. The problem is that it takes ages to load this list because I always get the whole content of the folder to a temporary list and then write the first 100 entries of that temporary list to my final list. Maybe it is faster if I could skip this extra step but I'm not sure if this is possible and if this would make any difference.

My code looks like this:

foreach (string Folder in FolderList)
{
    List<string> FileListTMP = new List<string>();
    FileListTMP.AddRange(Directory.GetFiles(Folder));
    int i = 0;
    while (i < FileListTMP.Count && i < 100)
    {
        FileList.Add(FileListTMP[i]);
        i++;
    }
}

It would be great if there is a solution to my problem because the number of folders is always growing and the loading times are already around 20 seconds long. :)

More info: At this time there are around 1500 folders. There are not more than 1000 files in a single folder. The average is about 500 files per folder.

Answer 1

Your posted code is directly equivalent to

FileList.AddRange(Directory.GetFiles(Folder).Take(100));

But the only thing that should take significant time in that piece of code is the GetFiles-call. Moving things between lists can be a problem if you are processing many millions of items, but should not be an issue when talking about hundreds of items. But you should use a performance profiler to verify this assumption, perhaps there is something completely different that takes time.

You could try changing .GetFiles to .EnumerateFiles since the later returns an enumerable rather than an array, so might be faster if you do not need a complete list of all files.

In my experience it might also be possible to get a performance speedup when listing files by using the native API rather than GetFiles. Something similar to this answer. But it does require a decent chunk of somewhat complex code.