Search code examples
c#large-data-volumeslarge-data

FileStream Seek fails on large files at second call


I'm working with large files , beginning from 10Gb. I'm loading the parts of the file in the memory for processing. Following code works fine for smaller files (700Mb)

 byte[] byteArr = new byte[layerPixelCount];
 using (FileStream fs = File.OpenRead(recFileName))
    {
        using (BinaryReader br = new BinaryReader(fs))
        {
            fs.Seek(offset, SeekOrigin.Begin);

            for (int i = 0; i < byteArr.Length; i++)
            {
                byteArr[i] = (byte)(br.ReadUInt16() / 256);
            }
         }
    }

After opening a 10Gb file, the first run of this function is OK. But the second Seek() throws an IO exception:

An attempt was made to move the file pointer before the beginning of the file.

The numbers are:

fs.Length = 11998628352

offset = 4252580352

byteArr.Length = 7746048

I assumed that GC didn't collect the closed fs reference before the second call and tried

    GC.Collect();
    GC.WaitForPendingFinalizers();

but no luck.

Any help is apreciated


Solution

  • I'm guessing it's because either your signed integer indexer or offset is rolling over to negative values. Try declaring offset and i as long.

    //Offest is now long
    long offset = 4252580352;
    
    byte[] byteArr = new byte[layerPixelCount];
    using (FileStream fs = File.OpenRead(recFileName))
    {
       using (BinaryReader br = new BinaryReader(fs))
        {
            fs.Seek(offset, SeekOrigin.Begin);
    
            for (long i = 0; i < byteArr.Length; i++)
            {
                byteArr[i] = (byte)(br.ReadUInt16() / 256);
            }
        }
    }