Redis: Transaction with multiple keys

Question

I'm using Spring Data Redis. In Redis the basic data models are

job: Hash that contains a job's data.

queue: List that contains job ids serving as a queue.

New job will be saved in job hash and it will be pushed to queue. We have multiple worker clients pooling the queue to consume new job id by popping the id and read details from hash.

Now I'm trying to work out a new feature that certain worker can only consume certain jobs, based on some flags within job data. The problem is worker will only know if it can consume the job after reading its details, but not at the moment of taking the id from the queue.

I originally thought I can put this sequence of operations into a transaction,

1. Peek the queue.
2. Read job details from hash and check if consumable.
3. If yes, take its id from queue, otherwise do nothing.

However this kind of transaction involves both queue and hash data. After reading Redis's transaction support I'm not sure if this is achievable. Please help to advise what approach should I take.

Answer 1

If you want to avoid the polling/peeking/race-conditions you could consider the following:

1. maintain your existing queue as the generic input queue and have a lightweight triage type worker (one is probably enough but you could have more for redundancy) pop items from this queue, using a blocking pop to avoid polling, and based on your job allocation logic push a new item onto a separate queue per worker-type

2. create multiple workers where each makes blocking pops from its own worker-type queue

One advantage of breaking your queue into multiple queues is that you get more visibility in your monitoring as you can check the length of each queue and see which types of workers are running slowly

It may not be suitable for production usage, but you may be interested in Disque, an in-memory distributed job queue, created by the same author as Redis (and based on Redis code).