Garbage collection can't keep up with Buffer creation and removal

Question

I have a method that runs every 2 seconds to capture a video stream to canvas and write it to file:

  function capture(streamName, callback) {
    var buffer,
      dataURL,
      dataSplit,
      _ctx;

    _ctx = _canvas[streamName].getContext('2d');
    _ctx.drawImage(_video[streamName], 0, 0);
    dataURL = _canvas[streamName].toDataURL('image/png');
    dataSplit = dataURL.split(",")[1];
    buffer = new Buffer(dataSplit, 'base64');

    fs.writeFileSync(directory + streamName + '.png', buffer);
  }

  setInterval(function() {
    // Called from here
    captureState.capture(activeScreens[currentScreenIndex]);
    gameState.pollForState(processId, activeScreens[currentScreenIndex], function() {
      // do things...
    });
  }, 2000);

Assuming _video[streamName] exists as a running <video> and _canvas[streamName] exists as a <canvas>. The method works, it just causes a memory leak.

The issue:

Garbage collection can't keep up with the amount of memory the method uses, memory leak ensues.

I have narrowed it down to this line:

buffer = new Buffer(dataSplit, 'base64');

If I comment that out, there is some accumulation of memory (~100MB) but it drops back down every 30s or so.

What I've tried:

Some posts suggested buffer = null; to remove the reference and mark for garbage collection, but that hasn't changed anything.

Any suggestions?

Timeline: https://i.sstatic.net/DOOpS.png https://i.sstatic.net/3NcXu.png

Allocation Profile: https://www.dropbox.com/s/zfezp46um6kin7g/Heap-20160929T140250.heaptimeline?dl=0

Just to quantify. After about 30 minutes of run time it sits at 2 GB memory used. This is an Electron (chromium / desktop) app.

SOLVED Pre-allocating the buffer is what fixed it. This means that in addition to scoping buffer outside of the function, you need to reuse the created buffer with buffer.write. In order to keep proper headers, make sure that you use the encoded parameter of buffer.write.

Answer 1

Matt, I am not sure what was not working with the pre-allocated buffers, so I've posted an algorithm of how such pre-allocated buffers could be used. The key thing here is that buffers are allocated only once for that reason there should not be any memory leak.

var buffers = [];
var bsize = 10000;

// allocate buffer pool
for(var i = 0; i < 10; i++ ){
    buffers.push({free:true, buf: new Buffer(bsize)});
}

// sample method that picks one of the buffers into use
function useOneBuffer(data){
    // find a free buffer
    var theBuf;
    var i = 10;
    while((typeof theBuf==='undefined')&& i < 10){
        if(buffers[i].free){
            theBuf = buffers[i];
        }
        i++;
    }

    theBuf.free = false;
    // start doing whatever you need with the buffer, write data in needed format to it first
    // BUT do not allocate
    // also, you may want to clear-write the existing data int he buffer, just in case before reuse or after the use.
    if(typeof theBuf==='undefined'){
        // return or throw... no free buffers left for now
        return;
    }
    theBuf.buf.write(data);
    // .... continue using

    // dont forget to pass the reference to the buffers member along because 
    // when you are done, toy have to mark it as free, so that it could be used again
    // theBuf.free = true;
}

Did you try something like this? Where did it fail?