HttpContent.CopyToAsync for large files

Question

Hope you're all doing well!

Lets say I'm downloading a file from an HTTP API endpoint and file size is quite large. API returns application/octet-stream i.e. HttpContent in my download method.

when I use

using (FileStream fs = new FileStrean(somepath, FileMode.Create))
{
    // this operation takes a few seconds to write to disk
    await httpContent.CopyToAsync(fs); 
}

As soon as the using statement is executed - I see the file created on the file system at given path, although it is 0 KB at this point, but when CopyToAsync() finishes executing, file size is as expected.

Problem is there's another service running which is constantly polling the folder where above files are saved and often times 0 KB are picked up or sometimes even partial files (this seems to be the case when I use WriteAsync(bytes[]).

Is there a way to not save the file on file system until its ready to be saved...?

One weird work around I could think of was:

using (var memStream = new MemoryStream())
{
    await httpContent.CopyToAsync(memStream);
    using (FileStream file = new FileStream(destFilePath, FileMode.Create, FileAccess.Write))
    {
        memStream.Position = 0;
        await memStream.CopyToAsync(file);
    }
}

I copy the HttpContent over to a MemoryStream and then copy the memorystream over to FileStream... this seems to have worked but there's a cost to memory consumption...

Another work around I could think of was to first save the files into a secondary location and when operation is complete, Move the file over to Primary folder.

Thank you in Advance,

Johny

Answer 1

I ended up saving the file into a temporary folder and when the operation is complete, I move the downloaded file to my primary folder. Since Move is atomic I do not have this issue anymore.

Thank you for those who commented!