node.js out of memory after file upload complete

Question

I have a web-server running on an embedded linux running on an armv7 SBC (Atmel SAMA5D27-SOM with 128MB RAM).

The firmware update process requires a .tar.gz file to be uploaded via this interface, then untarred/unzipped and further processes run on it.

After successful upload using busboy module, I noticed that the memory usage is still high (see attached image) even though the upload is streamed directly into a file (code below).

Further processes implemented with the child_processes after the file upload are then killed due to the system being out of memory.

I have implemented the --max-old-size-space-size flag to = 32MB as i've read this sets a lower threshold for when the GC starts.

I have tried various methods of using child_processes APIs including: 1. execSync 2. async exec (as described in node.js docs) 3. spawn with {detached: true}

/* imports omitted for brevity */

module.exports = async (req, res, next) => { // eslint-disable-line no-unused-vars
  try {

    /* File validation omitted for brevity */

    busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
      /* Further file validation omitted for brevity,
         File transfer debugging omitted for brevity */


      // Handle saving file stream to file
      const saveTo = path.join(updatesDir, path.basename(filename));
      file.pipe(fs.createWriteStream(saveTo));
    });

    busboy.on('finish', () => {
      return res.json({ updateFilename });
    });

    // This is required stream req progress back to browser
    return req.pipe(busboy);
  } catch (error) {
    console.error(error)
    next(error);
  }
}

I understand that node.js uses a garbage collector, however the memory taken for the file upload isn't freed for later processes to use.

Is there something wrong with how I am streaming the file upload to a file? Why is Node.js not releasing the memory? Is there a way to trigger GC manually in this instance?

Answer 1

Further testing reveal that the embedded linux system didn't include swap memory by default. I created a swap file and made it the same size as the systems RAM (128MB). Later I increased this to give extra overhead due to extra space availble on the SD Card.

This allows linux to manage the OOM issue automatically by moving non-critical memory usage onto disc. This process did slow down the upload by an extra few seconds but it did stop all OOM memory issues.