I'm trying to implement a task distribution system with RabbitMQ. I started with something like the code from this article: http://deontologician.tumblr.com/post/19741542377/using-pika-to-create-headers-exchanges-with - there is a headers exchange and multiple consumers' queues are bound to it with different header values.
Every message (task) has a header "env" that specifies an environment to run the task in. It might be necessary to make decisions based on more headers in the future. A consumer can provide more than one environment, so I bind his queue to the headers exchange multiple times with different header values.
This way, I can set up for example two consumers A an B. A provides environments "foo" and "bar" and B provides only "bar". Now when a task that requires environment "bar", it is delivered to both A and B, but I only want it to go to one of them (it doesn't really matter which one).
It seems that when a message is published that matches the headers of multiple consumers, it's delivered to all of them. However, I need each message to be delivered to exactly one consumer with matching headers. Is there any way to achieve this?
I can set up for example two consumers A an B. A provides environments "foo" and "bar" and B provides only "bar". Now when a task that requires environment "bar", it is delivered to both A and B, but I only want it to go to one of them (it doesn't really matter which one).
with your current setup, what you want will not be possible. all routing matches will receive a copy of the message.
what you can do, however, is change your configuration that you have a single "foo" queue and a single "bar" queue. then, you can have multiple consumers on the "foo" queue and multiple consumers on the "bar" queue.
In this scenario, when a single message is put into the "foo" and both consumer A and consumer B are listening, RabbitMQ will deliver the single message to only one of those consumers.
...
please keep in mind that it is impossible to 100% guarantee that a single message will only be handled exactly once. any error in network or consumer code could cause the message to be returned to the queue and processed again. because of this, your messages / consumers need to use idempotence to ensure processing the same message twice will not cause problems.