I have an application where I am persisting a large amount of binary data fairly quickly. I currently have two different data channels that can grow in size independently, so I store this using two files. However, these two files are closely coupled to each other (ID references that only make sense in the context of the other file), so it would be preferable, for the purposes of transport and proper linkage, to store/transport them as a single file.
Is there some standard way, at either the OS level or within the .NET BCL to construct two FileStream objects that can grow independently within the same underlying file?
I've considered implementing some sort of chunking/paging scheme, but it strikes me that this is basically duplicating what the file system already does under the hood, so I'd rather let the file system manage that, if it's possible.
Another alternative is to use a post-processing tool to combine the two files after recording completes. This is manageable, but not ideal.
I've experimented with some lightweight database options, as well, but they've proven to be too slow for what I'm doing.
An ideal solution to this problem would scale to more than two data streams in the same file, though two would be sufficient for my current purposes.
You can use NTFS alternate data streams (NTFS Alternate Data Streams - .NET) if you just want it to look like single file.
Note that many tools ignore alternate data streams during file operations, so be careful if you need to move such file around.