Search code examples
ccompressiongzip

Decompressing a gz file larger than available memory

I am on a device with 1 GB of memory and 16 GB of SSD disk space, but I want to use a 15 GB compressed PCAP file to stream IP data over the ethernet port.

Obviously, I cannot just decompress the file as I haven't enough disk space for that, and even if I did, the decompression would take too long. Does the gz file format support some form of streaming decompression so that I can stream the file without running out of disk space?

Solution

  • Does the gz file format support some form of streaming decompression so that I can stream the file without running out of disk space?

    Yes, it is possible to decompress gzip format incrementally. This is in fact Zlib's normal mode of operation when decompressing gzip.

    Or for a higher-level approach, the zcat orgunzip -c or gzip -dc command decompresses a gzipped file to its standard output.