Node worker process / cron job advice

Question

I have a database of items that I need to update — or rather just perform upon — every so often. I am using a message queue (Kue) to handle the concurrency of these jobs, but my process which goes about adding the jobs to the queue looks like this:

setInterval(function () {
  feed.find({}, function (error, foundModels) {
    jobs.create('update feeds', {
      feeds: foundModels
    }).save()
  })
}, 6000)

Is polling like this the best way to add the jobs to the queue, do you think? Or should each feed be on its own timer (for example every job will spawn another job 6 afters after it's finished)?

Answer 1

I usually do it the way you've done it. In your case, it always pushes jobs at 6 second intervals. This is fine so long as your jobs don't take more than 6 seconds. If your jobs take more than 6 seconds then you'll start to get a backlog and you'll need to increase resources to handle the larger load. It can be a problem if resource usage spikes and you're not around to adjust for the spike and you don't have automated processes in place (you should).

The alternative is to only call your function 6 seconds after the last call returns. You'd do that like so:

function update() {
  feed.find({}, function (error, foundModels) {
    jobs.create('update feeds', {
      feeds: foundModels
    }).save(function() {
      setTimeout(update,6000);
    });
  });
}
setTimeout(update, 6000);

I made the assumption that your .save method takes a callback like all good asynchronous libraries do. :-)