Search code examples
rabbitmqmessagingmessagebroker

RabbitMq: Dynamic number of consumers with direct bindings


I have the following scenario:

  • One Producer service
  • A dynamic amount of consumers services
  • Messages contain tasks with a specific product, so once consumer x handles a message of product y. In the future x should handle all messages of product y. Ideally the producer service should send all messages of product x on a queue which only consumer x reads from.
  • In order to divide workload evenly, there should be a way that once a new product needs to be managed, that next available consumer takes it.(I suppose a queue which all consumers are reading from)

My approach:

  • An exchange send new product jobs in a "newProduct" queue to which all the consumers are consuming from.
  • The consumer y that reads such a message notifies to the producer service (on a separate queue) that he is now in charge of product x.
  • The producer then sends all messages for product x to a queue proper to consumer y.
  • When a new consumer service z goes online, it notifies the producer service on a therefore specific queue that he is online such that the producer can create a binding in the exchange for z's proper queue.

Questions:

  • is my approach a good way to solve the problem, or am I missing rabbitmq solutions that would solve the problem in a less complicated way ?
  • How do I add a new queue during runtime to the exchange ?

Solution

  • An exchange send new product jobs in a "newProduct" queue to which all the consumers are consuming from.

    This looks good to me.

    The consumer y that reads such a message notifies to the producer service (on a separate queue) that he is now in charge of product x. This is also fine, I guess if producer did not receive notification that product X is taken care of it will need to do something. The producer then sends all messages for product x to a queue proper to consumer y.

    I'd send all messages for product X with the same routing key, like product-X. Which is what you probably mean here. I'd avoid telling producer who exactly handles the product-X now. For better separation of concerns and simplicity producers should know as less as possible about consumers and their queues and vice versa.

    When a new consumer service z goes online, it notifies the producer service on a therefore specific queue that he is online such that the producer can create a binding in the exchange for z's proper queue.

    You could do it this way, but I'd do it differently:

    When consumer goes online, it will create needed queues (or subscribe to existing queues) by itself.

    I see it like this:

    • Consumer comes online and subscribes to newProduct queue.
    • When received a message to handle product Z:
    • Creates a new queue for itself with binding key product-Z
    • Notifies producer that product Z is now being handled
    • Producer starts to send messages with routing key product-Z and they end up in Consumer's queue.

    Make sure your consumer has some High Availability, otherwise you may end up in the situation when your consumer started to handle some of the messages and then gone dead, while producer is continuing to send messages for now unhandled product.