Redis data not showing up until Node.js child_process.fork() completes

Question

I am forking a child process in a server app which does some repetitive CPU-bound work, a status value is pushed to Redis on every iteration.

The problem is that the status value does not show up on Redis until the child process completes, so I am only able to fetch the last status value.

I am polling for status value in a client app.

I'm using node_redis as Redis client.

I have verified the non-existence of status values before child process completion from redis-cli too.

parent (server app):

child_process.fork('child.js')

child (server app):

for (...) {
    //CPU-bound work
    redisClient.hset(key, field, value)
}

client app:

(function poll () {
    //wait
    redisClient.hget(key, field)
    poll()
})()

Answer 1

Try changing your child code to

async.eachSeries(data, function(element, next) {
   //do cpu bound work on element
   redisClient.hset(key, field, value, next); // wait for callback before starting next work
}, function(err) {
   console.log('done');
});

If you don't wait for the hset command callback what happens is the for loop will just keep going and won't get a chance to update the status. You might need to use a better async.* method for your needs.