The process cannot access the file because it is being used by another process at deletion

Question

I thought I solved this problem by closing and disposing my reader, but still in some cases the file was in use. Next I called the Garbage Collector, so the file will be released. This solves 99% of all the issues which will bring up this error. Code used:

        public override void Finish()
        {
            // Kill the reader!
            if (_reader != null)
            {
                _reader.Close();
                _reader.Dispose();

                // Make sure the server doesn't hold the file
                GC.Collect();
            }
            DeleteFile();
        }

Finish is called after a big process which will handle the file's content.

When I process a file with only 1 (or very few) line I sometimes get this error. It seems windows is to quick and DeleteFile(); fails. It is very hard for me to reproduce this error, but sometimes it just happens twice in a row.
This never occurs when I process a file with takes more then 2 seconds to process.
I cannot use a using because files can be GB's, and windows doesn't like it when its memory is getting too full. Also, this way the process performs way better.

Question:
Is there anything else I can do to prevent this error?

PS: Feel free to ask for more information.

EDIT:
Code to delete a file

        protected void DeleteFile()
        {
            // Delete the file
            if (FileName != null && File.Exists(FileName))
                File.Delete(FileName);
        }

Code to create the file

        protected void WriteFile()
        {
            // Prepare variables
            string path = Path.GetTempPath();

            path += "\\MyTempFile";

            // Modifiy path to avoid overwriting an existing file.

            path += ".csv";

            // Write the file to the temp folder
            using (FileStream fs = new FileStream(path, FileMode.Create))
            {
                fs.Write(MyFile, 0, MyFile.Length);
            }

            // Was the writing successfully completed?
            _FileName = File.Exists(path) ? path : null;
        }

Code to create the reader

        protected override void ReadFile()
        {
            if (FileName == null)
                WriteFile();

            // Read the lines
            _reader = new StreamReader(FileName, Encoding.Default, true);
            while (_reader.Peek() != -1)
            {
                TotalRows++;
                _reader.ReadLine();
            }

            // Read the lines
            _reader = new StreamReader(FileName, Encoding.Default, true);
        }

I use an abstract class to determine how the input should be read. With the following statement I'll loop through the content of the file.

 while (FileReader.NextRow(out currentRow, out currentRowNumber))
        {
               // Process current Row...
        }

The method NextRow() Looks like this

        public override bool NextRow(out List<object> nextRow, out int rowNumber)
        {
            if (RowNumber > TotalRows)
            {
                nextRow = null;
                rowNumber = 0;

                return false;
            }

            // Set the row number to return
            rowNumber = RowNumber;

            // Prepare the row
            nextRow = _reader.ReadLine().ExtensionThatProcessesTheRow();
            RowNumber++;

            return true;
        }

After the while loop closes I call the finish process. FileReader.Finish();

Answer 1

I think I found the problem...
this reader wasn't disposed before reseting it in ReadFile().

        _reader = new StreamReader(FileName, Encoding.Default, true);
        while (_reader.Peek() != -1)
        {
            TotalRows++;
            _reader.ReadLine();
        }

I changed it to.

            using (var reader = new StreamReader(FileName, Encoding.Default, detectEncodingFromByteOrderMarks: true))
            {
                while (reader.Peek() != -1)
                {
                    TotalRows++;
                    reader.ReadLine();
                }
            }