Search code examples
rabbitmqttlrabbitmq-shovel

Per message TTL behavior with shovels

Shovels

  • consumes messages from the queue,
  • re-publishes each message to the destination broker (using, by default, the original exchange name and routing_key when applicable).

I could not find any documentation what's the expected behavior for message TTLs when shovels are involved:

  • Does the time used for calculating TTL start when message is received at the source broker or at the destination broker? Or is it just valid for the first publish, that is at the source broker?

  • What happens if the expiration time elapses before the message reaches the destination broker?

Solution

  • So, I think you answered the question in the documentation you pasted in. All shovel does is move messages from one queue to another, re-publishing them in the process. It's going to preserve all original message properties, which theoretically includes the TTL property.

    That being said, I don't believe this is something you need to worry about.

    1. Message TTL starts when the queue receives the message. When the message is re-published, the clock resets on the new queue.
    2. Messages being transported by shovel will ideally spend no more than a few milliseconds in the initial queue, if they even end up there at all (a message queue with a consumer attached doesn't actually enqueue any messages under most conditions). So, the time spent in the first queue should be so small that it doesn't matter.
    3. Message lifetime should have a fair amount of tolerance for network transport, etc., so the activities of shovel are on par with the normal noise.

    If you find yourself in the situation where a large number of messages are accumulating in the queue before they can be shovel'd, then you might need to handle expiration in your application. There are other benefits and caveats to doing this, but you get a little finer-grained control overall.