Search code examples
azureazureservicebus

Delete messages from the DLQ after a specified period such as six months

Azure Service Bus entities (queues/topics) support a Time to Live (TTL). When the TTL passes the message expires. On expiry, the system deletes the message OR moves it to the Dead-Letter Queue (DLQ). Does Service Bus have another setting to delete messages from the DLQ after a specified period? For instance, to avoid passing size quotas, we might like to delete messages from the DLQ after six months.

See also:

Do messages in dead letter queues in Azure Service Bus expire?

https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-expiration?WT.mc_id=Portal-Microsoft_Azure_ServiceBus

Solution

  • Azure Service Bus doesn't have an expiration option on the dead-letter queues. This is likely intentional, as the system shouldn't just lose those messages but rather do something about them.

    Sometimes, monitoring all dead-letter queues for total size and whatnot is inconvenient. One option is to create a centralized DLQ. That will allow the following:

    1. Monitoring a single "dead-letter" queue.
    2. Receive messages from a single entity for processing.
    3. Keep the size under control by specifying a TTL on the queue.

    For example, let's say you've got two queues, test-dlq and test-dlq2. You'd configure those to auto-forward dead-lettered messages to a 3rd queue, test-dlq-all. With that, when you have messages that are received by test-dlq or test-dlq2 and dead-lettering,

    state 1

    Those messages will end up in the centralized "DLQ" queue (test-dlq-all).

    state 2

    The nice part is whenever you have messages auto-forwarded, you'll always know where they originally dead-lettered.

    For example, let's say you've got two messages, each from a different queue, ending up in test-dlq-all, the centralized "DLQ".

    state 3

    Inspecting its messages will reveal a system property, DeadLetterSource, stamped with the name of the queue it was dead-lettered initially in.

    state 4

    This solution lets you set TTL on the test-dlq-all queue and have messages auto-perged.

    Also, worth mentioning that it's possible to either set up dead-lettering with the centralized "DLQ" or get messaged dead-lettered as a result of failing processing that exceeds MaxDeliveryCount. For that reason, it is worth wither monitoring test-dlq-alls DLQ.