Queue messages in RabbitMQ are stacking and waiting on a single consumer while other consumers are available.
We use RabbitMQ as our messenger service. We use bunny to create connections and setup queues with passing messages.
In our rails setup using bunny, we are having an issue where we have one queue with 8 consumers listening on that queue for messages. When messages come in, ideally they should be round robin-ing the consumers example: 4 messages in the queue, consumer 1 picks up message 1, - consumer 1 is busy, consumer 2 picks up message 2, consumer 2 is busy, consumer 3 picks up message 3, and so forth.
However the issue we are having is, 4 messages in the queue, consumer 1 picks up message 1, consumer 1 busy, consumer 2-8 are available, but messages 2-4 are stacked in the queue waiting on consumer 1 to become available and process the messages.
I feel like i've done a bunch of research and just can't quite figure out how to stop messages from stacking and waiting on a single consumer.
Has anyone had experience in this, or have any ideas on how to fix this issue?
conn = Bunny.new(bunny[0])
conn.start
ch = conn.create_channel
q = ch.queue("#{record_queue_name}", :durable => true)
q.subscribe(:manual_ack => true, :arguments => {"x-priority" => 10}, :block => true) do |delivery_info, properties, payload|
ch.acknowledge(delivery_info.delivery_tag, false)
We are wanting anytime any messages get sent to RabbitMQ, the consumer picks them up on a first come first serve basis, not multiple messages stacked waiting for a busy consumer when others are available.
EDIT: How to reproduce: start 3 consumers at the same time. push through 6 messages - consumers 1 - 3 are now busy with 3 messages in the queue. restart 2 & 3, when 2 & 3 are listening again, 3 messages are still waiting in queue waiting on consumer 1. consumer 2 & 3 are still available.
Restart consumer 1, now 3 queue'd messages first come first server to newly restarted consumers 2 & 3.
I need messages to first come first server regardless if consumers restart.
RabbitMQ is working as intended.
Since your code does not set QoS / prefetch, RabbitMQ sends all six messages to your first consumer. Since that consumer takes time to acknowledge messages (simulated by a 45 second sleep in the code), those six remain in the "Unacked" state while your other two consumers don't have anything to work on. Restarting those other two consumers has no effect since all six messages are in "Unacked" waiting for an ack from the first consumer.
When you restart your first consumer, RabbitMQ detects the lost connection and enqueues the six messages to the "Ready" state, and delivers all six (most likely) to another consumer, and the problem repeats.
Please see this runnable code sample for how to set prefetch. With a prefetch of 1, RabbitMQ will at most deliver one message to a consumer and will wait for an ack before delivering another message to that consumer. In that way, messages will be distributed among your consumers.