Is there a way to read/write multiple structures or lines using Flatbuffers?

Question

I am using Flatbuffers as a way to store data and the meta-tags with the data. I am using Python and in order to simulate dictionaries, I have two table structures: One for dictionary entries and one to hold a vector of entries. Here is an example of the schema:

// Define dictionary structure

table tokenEntry{
  key:string;
  value:int;
}

table TokenDict{
  Entries:[tokenEntry];
}

root_type TokenDict;

I wish to write two dictionaries to a single file using Flatbuffers. I want to also read the dictionaries one at a time from the file, and not load both into memory at the same time. I am able to write both to file, one at a time. However, when I read from the file, I get both of the structures at once. The buffer holds all the data from the file. This is not what I want, because later I will have a much larger amount of data in the files. Is there a way to read in just one at a time?

As an example, if I were to use pickle structures, I can write multiple pickles to a file and read them back one at a time. i wish to do the same with Flatbuffers.

Thank you.

Answer 1

Best to write a file as a sequence of individual FlatBuffers, each prefixed with a size. You can do that in Python using FinishSizePrefixed (see builder.py).