Search code examples
c++flatbuffers

Is it possible to optimize flatbuffers serialization when constructing from native C++ structs?

I am using flatbuffers to serialize a bunch of class data using the "native struct" APIs from --gen-obj-api to populate a flatbuffer object as it was the simplest to use in my current code and I don't care as much about speed when writing. However, when deserializing on the read side I'm using the standard flatbuffer APIs as it doesn't require creating C++ objects and I can directly use the pointers for faster load times.

My questions:

  1. When serializing on the write side I am using the generated GeneratedClass::Pack() methods to serialize the native struct and all its submembers however there doesn't seem to be a way to do optimizations this way. By that I mean calls like CreateSharedString, CreateVectorOfSortedTables, etc. Ideally I would like to have those optimizations automatically done so I can take advantage of the "map-like" capabilities for looking up elements by key and reducing the storage requirements for repeated string names. Is there any way to do this using those Pack() APIs? The sorting I believe could be manually done by just sorting the std::vectors ahead of the Pack call, but I don't see any way around the SharedString optimization. It would be nice if there was a flag that could be set in the Pack call that would automatically perform those optimizations.

  2. When reconstructing std::vector during deserialization how would one construct from a flatbuffer::Vector? I know there are iterators but std::vectorbegin(), fbVec->end()) doesn't work because the string type is the flatbuffer string and there doesn't seem to be a way to easily construct using the c_str() option unless I manually loop through the strings.

Thanks!

Solution
    1. Yes, that is currently not possible, since those calls are automatically generated. We'd need to add attributes to fields like e.g. object_api: sorted or object_api: shared to hint at the code generator to use those alternative ways of constructing the data. You could make a PR for such functionality.
    2. You can't construct it this way because you're dealing with data that is fundamentally different from how the STL represents it. The only way to get nested data is thru UnPack which currently is only available on tables, though no reason we couldn't have a version that works on vectors or strings or other things. For now you'd have to write a manual loop that turns each element into a std::string as you go. If you're using C++17, a vector<string_view> would be way more efficient since it wouldn't allocate any strings, and reuse the existing data in the buffer.