Search code examples
.netcompression

Appending to a compressed file using GZipStream


I'm trying to use GZipStream to write some application traces (they tend to grow to huge sizes in production). So in this case i need the ability to open an existing file and append to it using GZipStream. All things seem to work well until we try to decompress the file. It seems that on decompression GZipStream reads only the first chunk of data and then behaves like it reached EOF (even if the file contains a whole lot more). Strange thing is that when opening the file using windows or Winrar all the data seems to be present and extracted properly. Has anyone encountered this issue before?


Solution

  • I've encountered same problem. The idea is to implement something like what http://zlib.net/pigz/ does.

    The idea is to remove last 8 bytes of old gzip chunk (the footer), extract CRC and size from the footer, then add some zeroes, then append new chunk and then recalculate source size and CRC bazed on old and new chunk's sizes and CRCs, and replace the resulting footer. The problem here is that I didn't find how to make valid sum CRC based on two parts CRCs. Also new chunk needs its header removed first.

    What pigz does is also sharing some of dictionary data between the chunks, and it does all the stuff described above, so you may look at the sources.