What to use as delimiter for stream of buffers

Question

I'm using flatbuffers to publish stream of objects in real time. One problem I met is how to separate between independent object/buffers. If I use some characters, like '\01' or '|' then there is no guarantee that those characters would not appear in the serialised buffer. What is the recommended way to do this?

while (more object) {
    flatbuffers::Offset<Msg> m = foo::CreateMessage(...);
    socket.send('|');
    socket.send(fbb.GetBufferPointer(), fbb.GetSize());
    socket.send('|');
}

Answer 1

Use a "length prefixed buffer" (see FlatBufferBuilder::FinishSizePrefixed and GetRootSizePrefixed) both on the sending and receiving end.

You are trying to use text base delimiting and parsing methods with a binary buffer. That will never work. You need to make sure your protocol is entirely in binary.