Search code examples
rabbitmqamqphigh-availability

AMQP/RabbitMQ - Process messages sequentially


I have one direct exchange. There is also one queue, bound to this exchange.

I have two consumers for that queue. The consumers are manually ack'ing the messages once they've done the corresponding processing.

The messages are logically ordered/sorted, and should be processed in that order. Is it possible to enforce that all messages are received and processed sequentially accross consumer A and consumer B? In other words, prevent A and B from processing messages at the same time.

Note: the consumers are not sharing the same connection and/or channel. This means I cannot use <channel>.basicQoS(1);.

Rationale of this question: both consumers are identicall. If one goes down, the other queue starts processing messages and everything keeps working without any required intervention.


Solution

  • Usually the point of a MQ system is to distribute workload. Of course, there are some situations where processing of message N depends on result of processing the message N-1, or even the N-1 message itself.

    If A and B can't process messages at the same time, then why not just have A or just B? As I see it, you are not saving anything with having 2 consumers in a way that one can work only when the other one is not...

    In your case, it would be best to have one consumer but to actually do the parallelisation (not a word really) on the processing part.

    Just to add that RMQ is distributing messages evenly to all consumers (in round-robin fashion) regardless on any criteria. Of course this is when prefetch is set to 1, which by default it is. More info on that here, look for "fair dispatch".