I'm looking for a distributed, reliable work queue. Where remote processes can take units of work from the queue when they are free.
I'm dealing with a distribution of quite heavy-weight tasks, where consumer microservices can only take one task at a time.
I've seen in examples that JGroups can provide reliable round robin message distribution. But I haven't managed to find any examples where the consumer can choose when to take a task.
Any advice would be greatly appreciated.
JGroups used to have a ReplicatedQueue, but I removed it some time ago. The reason was that I didn't like the semantics it provided: without total order (SEQUENCER), replicas could get out of sync. Also, network partitions were not handled in a meaningful way.
Besides, JGroups is more about reliable messaging and cluster management, and I didn't want to focus on building blocks too much.
However, it should be quite simple to create a replicated queue on top of jgroups-raft [1].