Search code examples
node.jsstreamgunzip

node.js how to stream 22GB gzipped file uncompress on the fly


I need to iterate over the lines in a 22GB gzipped file. Expanded, the file is 250GB.(It's the Google Freebase db dump)

I'm not too savvy with streams. Is there a way to stream the file and gunzip it as I go?

Here is going the other way:

var r = fs.createReadStream('file.txt');
var z = zlib.createGzip();
var w = fs.createWriteStream('file.txt.gz');
r.pipe(z).pipe(w);

Solution

  • Here is the reverse operation using zlib.createGunzip instead of zlib.createGzip:

    var r = fs.createReadStream('file.txt.gz');
    var z = zlib.createGunzip();
    var w = fs.createWriteStream('file.txt');
    r.pipe(z).pipe(w);