IOException reading a large file from a UNC path into a byte array using .NET

Question

I am using the following code to attempt to read a large file (280Mb) into a byte array from a UNC path

public void ReadWholeArray(string fileName, byte[] data)
{
    int offset = 0;
    int remaining = data.Length;

    log.Debug("ReadWholeArray");

    FileStream stream = new FileStream(fileName, FileMode.Open, FileAccess.Read);

    while (remaining > 0)
    {
        int read = stream.Read(data, offset, remaining);
        if (read <= 0)
            throw new EndOfStreamException
                (String.Format("End of stream reached with {0} bytes left to read", remaining));
        remaining -= read;
        offset += read;
    }
}

This is blowing up with the following error.

System.IO.IOException: Insufficient system resources exist to complete the requested 

If I run this using a local path it works fine, in my test case the UNC path is actually pointing to the local box.

Any thoughts what is going on here ?

Answer 1

I suspect that something lower down is trying to read into another buffer, and reading all 280MB in one go is failing. There may be more buffers required in the network case than in the local case.

I would read about 64K at a time instead of trying to read the whole lot in one go. That's enough to avoid too much overhead from chunking, but will avoid huge buffers being required.

Personally I tend to just read to the end of the stream rather than assuming that the file size will stay constant. See this question for more information.