Search code examples
javascriptnode.jsfile-iolazy-evaluation

Read a file one line at a time in node.js?


I am trying to read a large file one line at a time. I found a question on Quora that dealt with the subject but I'm missing some connections to make the whole thing fit together.

 var Lazy=require("lazy");
 new Lazy(process.stdin)
     .lines
     .forEach(
          function(line) { 
              console.log(line.toString()); 
          }
 );
 process.stdin.resume();

The bit that I'd like to figure out is how I might read one line at a time from a file instead of STDIN as in this sample.

I tried:

 fs.open('./VeryBigFile.csv', 'r', '0666', Process);

 function Process(err, fd) {
    if (err) throw err;
    // DO lazy read 
 }

but it's not working. I know that in a pinch I could fall back to using something like PHP, but I would like to figure this out.

I don't think the other answer would work as the file is much larger than the server I'm running it on has memory for.


Solution

  • Since Node.js v0.12 and as of Node.js v4.0.0, there is a stable readline core module. Here's the easiest way to read lines from a file, without any external modules:

    const fs = require('fs');
    const readline = require('readline');
    
    async function processLineByLine() {
      const fileStream = fs.createReadStream('input.txt');
    
      const rl = readline.createInterface({
        input: fileStream,
        crlfDelay: Infinity
      });
      // Note: we use the crlfDelay option to recognize all instances of CR LF
      // ('\r\n') in input.txt as a single line break.
    
      for await (const line of rl) {
        // Each line in input.txt will be successively available here as `line`.
        console.log(`Line from file: ${line}`);
      }
    }
    
    processLineByLine();
    

    Or alternatively:

    var lineReader = require('readline').createInterface({
      input: require('fs').createReadStream('file.in')
    });
    
    lineReader.on('line', function (line) {
      console.log('Line from file:', line);
    });
    
    lineReader.on('close', function () {
        console.log('all done, son');
    });
    

    The last line is read correctly (as of Node v0.12 or later), even if there is no final \n.

    UPDATE: this example has been added to Node's API official documentation.