Processing Folder With Multiple Files Using FileSystemWatcher and C#

Question

I have created a relatively simple windows app that watches a folder for files. When a new file is created into the folder the app (via FileSystemWatcher) will open the file and process the contents. Long story short, the contents are used with Selenium to automate a web page via IE11. This processing takes about 20 seconds per file.

The problem is if more than one file is created into the folder at roughly the same time or when the app is processing a file, FileSystemWatcher onCreated does not see the next file. So when processing completes on the first file the app just stops. Meanwhile there is a file in the folder that does not get processed. If a file is added after the onCreated processing is finished it works fine and processes that next file.

Can someone please guide me towards what I should be looking at to solve this? Excessive detail is very welcome.

Answer 1

FileSystemWatcher (as you've already noticed) is not reliable ,you will always have to add a "custom"/manual logic for missing files (also,note that you might see more than one event for the same file)

Below you can see a simple example with a "background" check for unprocessed file.
You could avoid the locks by using concurrent collections e.g BlockingCollection
You could also chose to process your files in parallel
I'm processing the file based on a timer but you could use your own strategy.
If you don't want to process the file ,in real time ,probably you don't even need the FileSystemWatcher

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;

namespace ConsoleAppDemo
{
    class Program
    {
        private static object lockIbj = new object();
        private static List<string> _proccessedFiles = new List<string>();
        private static readonly List<string> toProccessFiles = new List<string>();
        private static List<string> _proccessingFiles = new List<string>();
        private const string directory = @"C:\Path";
        private const string extension = @"*.txt";
        static void Main(string[] args)
        {
            FileSystemWatcher f = new FileSystemWatcher();
            f.Path = directory;
            f.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite
                             | NotifyFilters.FileName | NotifyFilters.DirectoryName;
            f.Filter = extension ;
            f.Created += F_Created;
            f.EnableRaisingEvents = true;

            Timer manualWatcher = new Timer(ManuallWatcherCallback, null, 0, 3000);

            Timer manualTaskRunner = new Timer(ManuallRunnerCallback, null, 0, 10000);

            Console.ReadLine();
        }

        private static void F_Created(object sender, FileSystemEventArgs e)
        {
            lock (lockIbj)
            {
                toProccessFiles.Add(e.FullPath);
                Console.WriteLine("Adding new File from watcher: " + e.FullPath);
            }

        }

        private static void ManuallWatcherCallback(Object o)
        {
            var files = Directory.GetFiles(directory, extension);
            lock (lockIbj)
            {
                foreach (var file in files)
                {
                    if (!_proccessedFiles.Contains(file) && !toProccessFiles.Contains(file) && !_proccessingFiles.Contains(file))
                    {
                        toProccessFiles.Add(file);
                        Console.WriteLine("Adding new File from manuall timer: " + file);
                    }
                }

            }
        }

        private static bool processing;
        private static void ManuallRunnerCallback(Object o)
        {
            if (processing)
                return;

            while (true)
            {
                //you could proccess file in parallel
                string fileToProcces = null;

                lock (lockIbj)
                {
                    fileToProcces = toProccessFiles.FirstOrDefault();
                    if (fileToProcces != null)
                    {
                        processing = true;
                        toProccessFiles.Remove(fileToProcces);
                        _proccessingFiles.Add(fileToProcces);
                    }
                    else
                    {
                        processing = false;
                        break;


                    }
                }

                if (fileToProcces == null)
                    return;

                //Must add error handling
                ProccessFile(fileToProcces);
            }
        }

        private static void ProccessFile(string fileToProcces)
        {
            Console.WriteLine("Processing:" + fileToProcces);
            lock (lockIbj)
            {
                _proccessingFiles.Remove(fileToProcces);
                _proccessedFiles.Add(fileToProcces);
            }
        }
    }
}