CEREAL failing to serialise - failed to read from input stream exception

Question

I found a particular 100MB bin file (CarveObj_k5_rgbThreshold10_triangleCameraMatches.bin in minimal example), where cereal fails to load throwing exception "Failed to read 368 bytes from input stream! Read 288"

The respective 900MB XML file (CarveObj_k5_rgbThreshold10_triangleCameraMatches.xml in minimal example), built from the same data, loads normally.

The XML file was produced by

    // {
        // std::ofstream outFile(base + "_triangleCameraMatches.xml");
        // cereal::XMLOutputArchive  oarchive(outFile);
        // oarchive(m_triangleCameraMatches);
    // }

and the binary version was produced by

    // { 
        // std::ofstream outFile(base + "_triangleCameraMatches.bin");
        // cereal::BinaryOutputArchive  oarchive(outFile);
        // oarchive(m_triangleCameraMatches);
    // }

Minimal example: https://www.dropbox.com/sh/fu9e8km0mwbhxvu/AAAfrbqn_9Tnokj4BVXB8miea?dl=0

Version of Cereal used: 1.3.0

MSVS 2017

Windows 10

Is this a bug? Am I missing something obvious? Created a bug report in the meanwhile: https://github.com/USCiLab/cereal/issues/607

Answer 1

In this particular instance, the "failed to read from input stream exception" thrown from line 105 of binary.hpp arises because the ios::binary flag is missing from the ifstream constructor call. (This is needed, otherwise ifstream will attempt to interpret some of the file contents as carriage return and linefeed characters. See this question for more information.)

So the few lines of code in your minimal example that read from the .bin file should look like this:

vector<vector<float>> testInBinary;
{
    std::ifstream is("CarveObj_k5_rgbThreshold10_triangleCameraMatches.bin", ios::binary);
    cereal::BinaryInputArchive iarchive(is);
    iarchive(testInBinary);
}

However, even after this is fixed there does also seem to be another problem with the data in that particular .bin file, as when I try to read it I get a different exception thrown, seemingly arising from an incorrectly encoded size value. I don't know if this is an artefact of copying to/from Dropbox though.

There doesn't seem to be a fundamental 100MB limit on Cereal binary files. The following minimal example creates a binary file of around 256MB and reads it back fine:

#include <iostream>
#include <fstream>
#include <vector>

#include <cereal/types/vector.hpp>
#include <cereal/types/memory.hpp>
#include <cereal/archives/xml.hpp>
#include <cereal/archives/binary.hpp>

using namespace std;

int main(int argc, char* argv[])
{
    vector<vector<double>> test;
    test.resize(32768, vector<double>(1024, -1.2345));
    {
        std::ofstream outFile("test.bin");
        cereal::BinaryOutputArchive oarchive(outFile, ios::binary);
        oarchive(test);
    }

    vector<vector<double>> testInBinary;
    {
        std::ifstream is("test.bin", ios::binary);
        cereal::BinaryInputArchive iarchive(is);
        iarchive(testInBinary);
    }

    return 0;
}

It might be worth noting that in your example code on Dropbox, you're also missing the ios::binary flag on the ofstream constructor when you're writing the .bin file:

/// Produced by:
// { 
    // std::ofstream outFile(base + "_triangleCameraMatches.bin");
    // cereal::BinaryOutputArchive  oarchive(outFile);
    // oarchive(m_triangleCameraMatches);
// }

It might be worth trying with the flag set. Hope some of this helps.