I need to iterate over the lines in a 22GB gzipped file. Expanded, the file is 250GB.(It's the Google Freebase db dump)
I'm not too savvy with streams. Is there a way to stream the file and gunzip it as I go?
Here is going the other way:
var r = fs.createReadStream('file.txt');
var z = zlib.createGzip();
var w = fs.createWriteStream('file.txt.gz');
r.pipe(z).pipe(w);
Here is the reverse operation using zlib.createGunzip
instead of zlib.createGzip
:
var r = fs.createReadStream('file.txt.gz');
var z = zlib.createGunzip();
var w = fs.createWriteStream('file.txt');
r.pipe(z).pipe(w);