I am writing msgpack-encoded data to a file. On writing, I am just using the fbuffer of the C API. As in (I striped all error handling for the example):
FILE *fp = fopen(filename, "ab");
msgpack_packer pk;
msgpack_packer_init(pk, fp, msgpack_fbuffer_write);
msgpack_pack_int(pk, 42);
// more data ...
How do I read this file back in? All the example I found assume that the data is in memory, however, my files are up to 5GB, it is not exactly a good idea to hold this in memory completely. Also I do not want to read in chunks myself. After all, I do not know how long the msgpack objects are, so chances are I end up with half an integer in my buffer.
Can msgpack's unpack somehow read from disk directly? Or is there some standard pattern to do this?
You might consider using "msgpack_unpacker" for that instead, which seems to be the official way that MessagePack implements a 'streaming' deserializer. Have a look at msgpack-c/example/c/lib_buffer_unpack.c
Regards, NiteHawk