Nodejs correct way to read endless stream

Question

I have web-app that works with realtime data through endless stream (or at least very long stream). I am using 'data' event to parse new chunk data on this stream. Problem is I have increasing memory consumption while this stream is alive. Since I do not call .read method for this stream all data is stored in stream buffer waiting to be processed. So, I want to ask, what can I do to reduce my memory consumption? Disabling inner buffering for this stream (I don't need old data), removing processed data from innerBuffer or mb working through .read method? Any advice appreciated.

.on('data', function(chunk) {
     parseString(chunk, function (err, result) {
     //...
     });
})

Answer 1

Using the data event is dangerous for this reason. It will dump data on you as fast as it can. In the face of asynchronous operations, this may be faster than you can process it, in which case your memory usage will steadily increase. You can fix this pretty easily by using pause/resume:

stream.on('data', function(chunk) {
  stream.pause();
  parseString(chunk, function (err, result) {
  //...
  stream.resume();
  });
})

You can also add parallelism by only pausing if there are already n parseString operations running.

You may also wish to look into new-style streams, which are usually preferred.