Search code examples
bashredisredis-clijob-queue

Can I implement a redis job queue with plain bash and redis-cli?

I am relatively new to redis, and I am looking to implement a job queue as a simple list of job ids (and nothing else in redis).

I've searched around, but couldn't find examples of how to do this in plain shell interface (redis-cli). The examples are mostly using some programming languages such as NodeJS, GO, Java, and Python. But I don't want to be bound to one of these languages. To reduce dependency, I would like to implement the workqueue in plain bash/shell, since my application need is not that complex.

Specifically, I have a pre-defined list of jobs, each with an integer id. I intend to set up a redis-server and multiple worker nodes that each takes/deletes job ids one at a time from the redis queue/list (and then processes it).

Based on reading the linked articles, I guess the steps are:

  1. initialize the list by RPUSH-ing all job ids to the redis-server.

  2. each node somehow takes and deletes a job id from the list, compute it, and then save a small amount of result (as a json encoded value) to another data structure in redis.

My questions are:

Can this be done using plain bash scripts and redis-cli, or is there a reason why a full-fledged programming language is required?

How can redis-cli instances among the nodes handle concurrent use of the queue in Point 2? That is, if one node is deleting a job id, make sure the other nodes in the cluster do not read/see the same id again. What commands are appropriate for this?

By the way, this is all with Ubuntu 22.04. There is no parallelism or expiration, and each node sequentially takes one job, completes it, and takes next, until there is no more jobs to take. Also, I'd use a single redis-server and not a cluster of redis-servers, for simplicity.

Solution

  • This should be simple enough. Redis is single-threaded and all commands are atomic.

    To push things to the List use LPUSH:

    $ redis-cli LPUSH foo 1
(integer) 1

    To pop things from the List use RPOP:

    $ redis-cli RPOP foo
1) "foo"
2) "1"

    If you want to wait for something if the List is empty, just use BRPOP instead:

    $ redis-cli BRPOP foo 60
1) "foo"
2) "1"

    Since each command is atomic, only a single call to RPOP or BRPOP will receive any given item popped to the List.

    You can test this out easily enough by trying a couple of BRPOP calls in a couple terminal windows and then adding something to the List using LPUSH. Only one terminal will get the popped value.