Is there any way to repair files corrupted by not properly closing boost::archive::binary_oarchive?

Question

I ran a large processing job which produced lots of binary files as output. I think I realize now that my output data files got corrupted because I did not properly flush or close the boost::archive::binary_oarchive object before moving the data files to remote storage. I'm wondering if there is any way to repair the output data files by appending them with some special EOF thing, or whether I'm out of luck and I have to re-run the expensive job?

More specifically, my processing job dumps the binary data like so:

   void dumpStuff() {
        // some code
        std::ofstream ofs(localFileName);
        boost::archive::binary_oarchive boa(ofs);
        boa << *data;   

        if (uploadToRemote) {
            // code that uploads files to remote store
            // does not run when I tested locally
        }
    }

What I think happens is that when I tested locally (and did not upload to remote), the boa object goes out of scope at the end of the dumpStuff function and so it's destructor gets called, which properly flushes the stream and closes the file. When uploading to remote store, however, the upload happens before the oba's destructor is called, so it is my belief that the stream was not properly flushed, resulting in a corrupt file. When I fetch the corrupted file from the store and attempt to load with boost::archive::binary_iarchive, I get an InputStreamError.

I know I can fix the problem by adding some braces around the boa stuff to force it to go out of scope before uploading to remote, however, this will only solve my problem if I re-run the big expensive job. So, my question is, is there some simple way to just append something to the end of my corrupt files to de-corrupt them? Some kind of EOF signal?

Answer 1

There may not be. Then again most of it will surely depend on the flushing behaviour of the underlying stream you used.

This is a one-off problem that only you have, so you will have to make a solution.

• One way would be to look at the source code to figure out exactly what actions would be skipped due to the missing close. And then either compensate the missing input OR make the input-archive implementation more tolerant for corrupt/missing tails.

• The other approach would be to use your own code WITH the flaw to write an archive, and then write the SAME archive but with the error fixed.

  Just look at the difference in a hex editor. You may be lucky and find that the data missing from the archive is fixed. If so, just append it to any corrupt input stream and be glad. More likely you will have some (simple) variable data, like a checksum or a total size. In that case either try to generate the missing data, or hack the input-stream implementation to detect the required checksum.

CAVEAT: All of these suggest meddling with undocumented details, there will not be support, reliability depends solely on your own accuracy.

If you choose to "fake" checksums, be aware of the fact that it thwarts any builtin error-detection, so you might still read unreliable data (in case there was data corrupted in sotrage/transit)