Use Asynchronous IO better

Question

I am really new to JS, and even newer to node.js. So using "traditional" programming paradigms my file looks like this:

var d = require('babyparse');
var fs = require('fs');

var file = fs.readFile('SkuDetail.txt');

d.parse(file);

So this has many problems:

1. It's not asynchronous
2. My file is bigger than the default max file size (this one is about 60mb) so it currently breaks (not 100% sure if that's the reason).

My question: how do I load a big file (and this will be significantly bigger than 60mb for future uses) asynchronously, parsing as I get information. Then as a followup, how do I know when everything is completed?

Answer 1

You should create a ReadStream. A common pattern looks like this. You can parse data as it gets available on the data event.

function readFile(filePath, done) {
    var 
        stream = fs.createReadStream(filePath),
        out = '';

    // Make done optional
    done = done || function(err) { if(err) throw err; };

    stream.on('data', function(data) {
        // Parse data
        out += data;
    });

    stream.on('end', function(){
        done(null, out); // All data is read
    });

    stream.on('error', function(err) {
        done(err);
    });
}

You can use the method like:

readFile('SkuDetail.txt', function(err, out) {
    // Handle error
    if(err) throw err;

    // File has been read and parsed
}

If you add the parsed data to the out variable the entire parsed file will be sent to the done callback.