Missing file entries when multiple threads-generated processes copy files to same directory C#

Question

I am trying to copy large number of files from various directories on different hosts to a single directory. Since it turned out to be quite time consuming, and I have to run this program multiple times, I changed the code to leverage the help of multi-thread, with each thread doing XCOPY through a process.

However, I found out that when I am doing multi-thread copying, some files (less than 100, varies each time) are not getting copied. I have not figured whether it's because of multiple processes intervening each other when XCOPYing to the same directory, or it's more of a XCOPY issue (not sure how well it supports multi-process calling on same destination).

List<Thread> threadList = new List<Thread>;

foreach(FileEntry fileEntry in fileEntries)
{
    Thread thread = new Thread(() => {
        //full path of file, like \\host\directory\directoryB\fileA.txt
        string filePath = fileEntry._filePath;

        //name of file, like fileA.txt
        string file = fileEntry._file;

        //dumpDirectory is where I want to copy all the files to
        string destPath = Path.Combine(dumpDirectory, file);

        //each file here is either a directory or a real file, bad naming convention I know...(open to suggestions if any)
        string fileType = File.Exists(filePath) ? "f" : "d";

        using (Process process = new Process())
        {
            process.StartInfo.CreateNoWindow = true;
            process.StartInfo.UseShellExecute = false;
            process.StartInfo.FileName = "cmd.exe";

            process.StartInfo.Arguments = string.Format(@"/C echo {0} | XCOPY /E /C /H /R /K /O /Y {1} {2}", fileType, filePath, destPath);

            Console.WriteLine("copying {2} [{0}] to destination [{1}]", filePath, destPath, string.Compare(fileType, "f") == 0 ? "file" : "directory");
            process.Start();
            process.WaitForExit();

            if (process.ExitCode != 0)
            {
                Console.WriteLine("encountered problems when copying file [{0}]", filePath);
            }
        }
    });

    thread.Start();
    threadList.Add(thread);
}

foreach(Thread thread in threadList)
{
    thread.Join();
}

---------------------------Bug Fix As Following -------------------------------

So as suggested by replies below (thanks guys for the quick replies, saved my weekend ^_^), I redirected the process output and found out that the issue is "Sharing violation \n Unable to create directory ". xcopy is having issues when copying multiple files into the same directory (various xcopy processes seem to create a directory at the same time when they all detect that the directory does not exist on the system).

Changing the manual thread management to Parallel.Foreach solved the problem and made the code look nicer (though I have not figured out why it is not creating the same problem)

Or, a dirtier fix is to wrap up the process.Start() with a EventWaitHandle. I used different EventWaitHandle instances based on the destination directory rather than using one handle because the other way will defeat the purpose of using multi-process

//name the wait handle based on destination value
EventWaitHandle destWaitHandle = new EventWaitHandle(true, EventResetMode.AutoReset, string.Format("waitHandle{0}", destPath.GetHashCode()));

//only trigger the wait handle lock if the file being copied from (and to) is a directory. Based on the error log info XCopy seemed to never have issues with copying individual files
//and the program worked fine if I only put the lock at directory level
if (string.Compare(fileType, "d") == 0)
{
    destWaitHandle.WaitOne();
}

process.Start();

//this line was added to write error messages into respective files during the process' execution
process.BeginErrorReadLine();
process.WaitForExit();

if (string.Compare(fileType, "d") == 0)
{
    destWaitHandle.Set();
}

Answer 1

First of all, I suggest you to switch to the Task-oriented code, as the creation on threads in such number could lead to downgrade the performance because of the context switches.

You easily can do this with Parallel extensions, like this:

Parallel.ForEach(fileEntries, fileEntry => { // YOUR code here };

Second, you can investigate the error occured in the XCOPY process by examing the StandardError property of the Process object, as @x... suggested, something like this (you also have to redirect it for ability to read it):

process.StartInfo.RedirectStandardError = true;
// ...
if (process.ExitCode != 0)
{
    Console.WriteLine("encountered problems when copying file [{0}]", filePath);
    Console.WriteLine(process.StandardError.ReadToEnd());
}