Search code examples
javaspringspring-integrationgoogle-cloud-pubsubspring-cloud-gcp

Spring Cloud GCP PubSub DLQ (dead letter queue) using Spring Integration


I'm currently integrating PubSub using the following approach https://docs.spring.io/spring-cloud-gcp/docs/1.2.4.BUILD-SNAPSHOT/reference/html/#spring-integration

Basically I'm subscribed to a topic, I receive the message and do something with it.

I would like to, in case of any error scenario of my business rules (like not accept the message because X,Y,Z reason) send it to a dlq.

I see that just a month and a half ago Google came out with DLQ for PubSub: https://cloud.google.com/pubsub/docs/dead-letter-topics

But I'm not sure what the right way to integrate it with the spring integration approach should be.


Solution

  • The built-in DLQ support will work automatically -- all you have to do is turn on Dead Lettering (through add/edit subscription screen in Cloud Console), setting the "Maximum delivery attempts" field to the number of attempts. After your application nack()s a message this number of times, Cloud Pub/Sub will redirect it to the topic you set up as the DLQ.

    The built-in support works great for environmental, retryable causes of message delivery failure. However, it has a minimum of 5 retries before it will send the message to DLQ. In your case of business rules validation, you may prefer to mimic DLQs with a custom topic and Spring Integration redirection, since after failing on attempt #1, the message won't suddenly become valid, and the remaining attempts #2 - #5 are a waste of resources.

    To mimic the DLQ that redirects after a single failure, you would create a new topic, let's call it validation-dlq, wire it up as a Spring Integration channel, run custom validation checks on each message, and if a message fails, ack() the original message on the source subscription, and publish the message to the validation-dlq topic instead.