How to create a queue to store incoming files added to a folder, when more than one file is dropped in at a time?

Question

I made a service which monitors a folder to see when a new file is added. When a new file is added, the file is broken down by line (each line is stored in a list), and the data stored in each line is then encoded into a QR code image. The QR image is then saved to a new folder. This whole process works BUT when I drop more than one file into a folder at the same time, my service only encoded the lines of one of the files.

Issue: How can I create a queue to store newly added files (when more than one file is dropped in at a time) in a list which I can pull from once it encodes the current file in the queue??

First I am using System.Collections.Concurrent queue to make a method to grabdata in the queue.

Watcher Methods: Start the service/file system monitor

public void Start()
{
    newfolders = ConfigurationManager.AppSettings["parsefiles"];
    pathtomonitor = ConfigurationManager.AppSettings["pathtomonitor"];

    pathforpaths = newfolders;

    System.IO.Directory.CreateDirectory(pathforpaths);
    Thread.Sleep(2000);

    pathforprocess = newfolders + @"\processdata";

    System.IO.Directory.CreateDirectory(pathtomonitor);
    System.IO.Directory.CreateDirectory(pathforprocess);

    FileSystemWatcher watcher = new FileSystemWatcher(pathtomonitor);

    watcher.NotifyFilter = NotifyFilters.Attributes | NotifyFilters.FileName;

    watcher.EnableRaisingEvents = true;
    watcher.IncludeSubdirectories = false;

    //add event handlers
    watcher.Created += watcher_Created;
}

Get File added to folder and create QR code

private static void watcher_Created(object sender, FileSystemEventArgs e)
{
    if (e.ChangeType == WatcherChangeTypes.Created)
    {

        if (DateTime.Now.Subtract(_lastTimeFileWatcherEventRaised)
            .TotalMilliseconds < 1000)
        {
            Console.WriteLine("Short time between previous event and change event");
            return;
        }

        Console.WriteLine("file: {0} changed at time:{1}", e.Name,
            DateTime.Now.ToLocalTime());

        Thread.Sleep(1000);
        if (GetAccessLoop(pathtomonitor + "\\" + e.Name))
        {
            parsefile(pathtomonitor + "\\" + e.Name);
            Console.WriteLine("worked");
        }
        else
        {
            //writetologs("Was not able to get access to" + e.Name);
            Console.WriteLine("didnt work");
        }

        _lastTimeFileWatcherEventRaised = DateTime.Now;
    }
}

**ISSUE: **Only one of the files gets stored in the string not both. I want all files added to the folder to be stored so I can pull the files from the list one at a time and delete them once they are done being encoded.

How do I store all of the files detected by filesystem watcher so I can save files that have been dropped in at the same time?

Would I use a memory stream?????

Answer 1

Sleeping in the event handler is a bad idea. Event handlers should be as quick as possible. You want the event handler to make note of the new file (for example, by adding the file's name or path to a thread-safe collection such as a ConcurrentQueue) then exit. You then want a separate task to process entries in that collection as necessary for your program.

Here's a simplified example:

async Task Main()
{
    var pathtomonitor = @"C:\Temp\FileWatcherExample";
    
    System.IO.Directory.CreateDirectory(pathtomonitor);

    FileSystemWatcher watcher = new FileSystemWatcher(pathtomonitor);

    watcher.NotifyFilter = NotifyFilters.Attributes | NotifyFilters.FileName;

    watcher.EnableRaisingEvents = true;
    watcher.IncludeSubdirectories = false;

    //add event handlers
    watcher.Created += watcher_Created;
    
    // As programmed, this will await forever since there 
    // isn't a mechanism to end ProcessNewFiles()
    await ProcessNewFiles();
    
}

ConcurrentQueue<string> createdFiles = new ConcurrentQueue<string>();

private async Task ProcessNewFiles()
{
    while (true) // In production code we'd use a cancellation token to exit
    {
        while (createdFiles.TryDequeue(out var path))
        {
            Console.WriteLine($"Processing {path}");
        }
        // This is more efficienty handled with Channels, but we'll
        // introduce a small delay here to keep the code example
        // simple
        await Task.Delay(100);
    }
}

private void watcher_Created(object sender, FileSystemEventArgs e)
{
    if (e.ChangeType == WatcherChangeTypes.Created)
    {
        Console.WriteLine($"New file: {e.FullPath}");
        createdFiles.Enqueue(e.FullPath);
    }
}

I am a fan of the suggestion in the other answer, to move files into an "Archived" folder once they have been processed.