RabbitMQ - Move messages before deleting a queue

Question

Using RabbitMQ 3.7.16, with spring-amqp 2.2.3.RELEASE.

Multiple clients publish messages to the DataExchange topic exchange in our RabbitMQ server, using a unique routing key. In the absence of any bindings, the exchange will route all the messaged to the data.queue.generic through the AE.

When a certain client (client ID 1 and 2 in the diagram) publishes lots of messages, in order to scale the consumption of their messages independently from other clients, we are starting consumers and assign them to only handle a their client ID. To achieve this, each client-consumer is defining a new queue, and it binds it to the topic exchange with the routing key events.<clientID>.

So scaling up is covered and works well.

Now when the messages rate for this client goes down, we would like to also scale down its consumers, up to the point of removing all of them. The intention is to then have all those messages being routed to the GenericExchange, where there's a pool of generic consumers taking care of them.

The problem is that if I delete data.queue.2 (in order to remove its binding which will lead to new messages being routed to the GenericExchange) all its pending messages will be lost.

Here's a simplified architecture view:

It would be an acceptable solution to let the messages expire with a TTL in the client queue, and then dead letter them to the generic exchange, but then I also need to stop the topic exchange from routing new messages to this "dying" queue.

So what options do I have to stop the topic exchange from routing messages to the client queue where now there's no consumer connected to it?

Or to explore another path - how to dead letter messages in a deleted/expired queue?

Answer 1

If the client queue is the only one with a matching binding as your explanation seems to suggest, you can just remove the binding between the exchange and the queue.

From then on, all new messages for the client will go through the alternate exchange, your "generic exchange", to be processed by your generic consumers.

As for the messages left over in the client queue, you could use a shovel to send them back to the topic exchange, for them to be routed to the generic exchange.

This based on the assumption the alternate exchange is internal. If it's not internal, you can target it directly with the shovel.

As discussed with Bogdan, another option to resolve this while ensuring no message loss is occuring is to perform multiple steps:

• remove the binding between the specific queue and the exchange
• have some logic to have the remaining messages be either consumed or rerouted to the generic queue
  • if the binding removal occurs prior to the consumer(s) disconnect, have the last consumer disconnect only once the queue is empty
  • if the binding removal occurs after the last consumer disconnect, then have a TTL on messages with alternate exchange as the generic exchange
• depending on the options selected before, have some cleanup mecanism to remove the lingering empty queues