Search code examples
serializationflatbuffers

Size of Serialized data is not reducing using flatbuffer


I have written following fbs file

namespace testing;

table polygon {
    x : double;
    y  : double;

}

table layer {
    layer_name : string;
    polygons : [polygon];
}

root_type layer;

My plan is to serialize approx 5 Million coordinates and dump it into one file. Problem is what I see is the number of bytes is increased compared to what I was expecting. I am expecting it should be arounf (5M* 16) bytes. But the size what I am getting is 140000032 bytes

Here is the java code which I am using for dumping serialize data into a file.

FlatBufferBuilder fbb = new FlatBufferBuilder(0);
    String s = "Product1";
    int str = fbb.createString("layer1");
    int size = 1 * 5 * 1000000;
    int[] offset = new int[size];
    int cur = 0;

    for (double i = 0; i < size; i++) {

        fbb.startTable(2);
        polygon.addX(fbb, i);
        polygon.addY(fbb, i);
        offset[cur++] = polygon.endpolygon(fbb);
    }

    int arrayoffset = layer.createPolygonsVector(fbb, offset);
    layer.startlayer(fbb);

    layer.addLayerName(fbb, str);

    layer.addPolygons(fbb, arrayoffset);
    int bla = layer.endlayer(fbb);

    fbb.finish(bla);
    ByteBuffer bf = fbb.dataBuffer().duplicate();

    File myfile = new File("/tmp/test.dat");

    try {
        FileChannel channel = new FileOutputStream(myfile).getChannel();
        channel.write(bf);
        channel.close();

    } catch (FileNotFoundException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }

Please let me know if I am doing something wrong here


Solution

  • change table polygon into struct polygon. table is extensible and has some fixed overhead per element, and also is referred to by the vector over an offset. A struct is not extensible (which seems fine for an xy pair), and has to be serialized inline (see example in the tutorial), and will give you the size you expect.