FlatBuffers: How to write giant files

Question

I'm experimenting with FlatBuffers to store OpenStreetMap data. So, this would be 50GB.

Since usually everything is kept in memory, how is it possible (if at all) to sequentially write data into a file?

I have a feeling this is not quite where FlatBuffers is good for.

Answer 1

There currently is no way to create a single FlatBuffer without having it all in memory at once. The only way to do it is to instead write out a chain of (length-prefixed) smaller independent FlatBuffers.