File Chunking Performance in C#

Question

I am trying to empower users to upload large files. Before I upload a file, I want to chunk it up. Each chunk needs to be a C# object. The reason why is for logging purposes. Its a long story, but I need to create actual C# objects that represent each file chunk. Regardless, I'm trying the following approach:

public static List<FileChunk> GetAllForFile(byte[] fileBytes)
{
  List<FileChunk> chunks = new List<FileChunk>();
  if (fileBytes.Length > 0)
  {
    FileChunk chunk = new FileChunk();
    for (int i = 0; i < (fileBytes.Length / 512); i++)
    {
      chunk.Number = (i + 1);
      chunk.Offset = (i * 512);
      chunk.Bytes = fileBytes.Skip(chunk.Offset).Take(512).ToArray();

      chunks.Add(chunk);
      chunk = new FileChunk();
    }
  }
  return chunks;
}

Unfortunately, this approach seems to be incredibly slow. Does anyone know how I can improve the performance while still creating objects for each chunk?

thank you

Answer 1

I suspect this is going to hurt a little:

chunk.Bytes = fileBytes.Skip(chunk.Offset).Take(512).ToArray();

Try this instead:

byte buffer = new byte[512];
Buffer.BlockCopy(fileBytes, chunk.Offset, buffer, 0, 512);
chunk.Bytes = buffer;

(Code not tested)

And the reason why this code would likely be slow is because Skip doesn't do anything special for arrays (though it could). This means that every pass through your loop is iterating the first 512*n items in the array, which results in O(n^2) performance, where you should just be seeing O(n).