node.js - generating 1000s of images causing heavy load

Question

i am new to node.js. I want to generate a number of images by imagick,gmagick for testing purposes. The images should be name from 000001.jpg to 0x.jpg. For the later testing i need about 50.000 images.

var async = require('async');
var gm = require('gm')
            .subClass({ imageMagick: true }); // Enable ImageMagick integration.


var number = parseInt(process.argv[2]) || 20;
console.log("Generating " + number + " images");

var pad = "000000";
var img_names = [];
for (i=1;i<=number;i++) {
    var str = "" + i;
    var padded_str = pad.substring(0, pad.length - str.length) + str;
    img_names.push(padded_str+".jpg");
}

async.forEach(img_names, function (item, callback){ 
    console.log(item); // print the key
    gm(800, 600, "#ddff99f3").font("Helvetica.ttf", 80)
    .drawText(150, 300, "# "+item)
    .write("./"+item, function (err) {
      if(err) {
        console.error(err);
        process.exit(1);
      }
    }); 
    callback(); // tell async that the iterator has completed

}, function(err) {
    console.log('iterating done');
});

The proble is, when using a higher amout of images, the ubuntu machine starts a huge number of "convert" processes and gets into heavy load.

Could anyone point me in the right direction? Kind regards, Robert

Answer 1

You need to move the callback() function into the write function's callback, as the iteration has just forked gmagick, and not actually waited for it to finish. When it finishes, it calls its own callback - this is the end of the iteration, so it's where the forEach() function's callback should be called.

Moving callback() as above will block the async.forEach() until the gm() function has finished.

If you have a multicore machine you can run this function in parallel, approximately one worker process per core/hyperthread is a sensible default. Swap forEach for eachLimit() and set parallelism to 8:

async.eachLimit(img_names, 8, function (item, callback){ 
    console.log(item); // print the key
    gm(800, 600, "#ddff99f3").font("Helvetica.ttf", 80)
    .drawText(150, 300, "# "+item)
    .write("./"+item, function (err) {
      if(err) {
        console.error(err);
        process.exit(1);
      }
      callback(); // tell async that the iterator has completed
    }); 

}, function(err) {
    console.log('iterating done');
});