How can one ensure that FileStream is disposed of after all WriteAsync() on the current file are complete?

Question

I have the following event delegate that basically takes in an array of bytes, and appends it to a filestream. Once it has written 1,000 times, which is tracked by a local variable counter, I want to close the current filestream and open a new one.

I read through the documentation of FileStream, and it suggested using FileStream.WriteAsync() for better performance.

public void WriteData(byte[] data)
{
   counter++;
   if (counter == 1000)
   {
         counter = 0;

         // Close the current filestream and open a new one.
          filestream.Dispose();
          filestream = new FileStream(this.outputPath, FileMode.Create,
            FileAccess.Write, FileShare.None, 4096, true);
   }

   filestream.WriteAsync(data, 0, data.Length);
}

However, in the function above, my hypothesis is that it might be the case that all the WriteAsync() calls have not completed before I call filestream.Dispose(). Is there a way to ensure that I only Dispose() after all my WriteAsync() calls have been completed? Note that this event delegate is being called 1,000 - 2,000 times per second in sequence, and WriteAsync is copying 240 KB per call to a SSD.

One solution that I can think of is instead of immediately disposing of each Filestream immediately, I can store it in an array of Filestream and then dispose of them once I am finished with the entire data writing process and no more events are firing. Would that work? Even then, how can I effectively "wait" until all the WriteAsync() calls have completed?

Answer 1

While not an answer to the exact question, based on the follow-up in comments I'd suggest you do the following:

• Create a ConcurrentQueue in whatever is firing the events
• On each event, add to that queue
• Create a separate thread that services the queue, dequeuing events and writing them to disk. You can then control exactly how frequently you want to open, write, rename, etc., and you have no thread concurrency issues on the file activity to deal with.
• Depending on your requirement, in your writer thread you could just poll the queue every second or so, or you could have a more sophisticated approach using WaitHandles to signal when data is ready for writing, when the queue is empty, when it's got N items in it, etc. As you're accumulating data so frequently, perhaps polling is fine as you'll almost always find data to write.
• You obviously need to handle closing the app cleanly - e.g. stop writing from events, single to the writing thread to stop, have it flush the last items to disk, wait for it to stop.

It's more work, but you get:

• guaranteed order
• almost no time spent in the event handler
• no concurrency issues (except for managing setup and tear down)
• delays in writing - e.g. due to OS, disk, network - do not block the main app
• a risk of losing event data if you lose power or the app aborts (you will have this with any solution that doesn't wait for the data to be written after each event)

There are probably out-the-box solutions out there for this (using a buffered log4net Appender comes to mind); this is if you want to roll your own.

You could do this with 'standard' threading (i.e. Thread), or you could just create your writing thread as a Task.