Search code examples
spring-bootamazon-sqs

AWS SQS, DLQ in spring boot

How do I add a DLQ configuration to my SQS configuration? I'm not sure how to integrate a DLQ with my existing queue. I'm using aws messaging and not JMS, so my annotation would be @SQSListener for my listener method. I have a config class that has following

@Bean
public SimpleMessageListenerContainer messageListenerContainer(AmazonSQSAsync amazonSQSAsync) {
    SimpleMessageListenerContainerFactory factory = new SimpleMessageListenerContainerFactory();
    factory.setAmazonSqs(amazonSQSAsync);
    factory.setMaxNumberOfMessages(10);
    SimpleMessageListenerContainer simpleMessageListenerContainer = factory.createSimpleMessageListenerContainer();
    simpleMessageListenerContainer.setQueueStopTimeout(queueStopTimeout*1000);
    simpleMessageListenerContainer.setMessageHandler(messageHandler(amazonSQSAsync));
    return simpleMessageListenerContainer;
}

@Bean
public QueueMessageHandler messageHandler(AmazonSQSAsync amazonSQSAsync) {
    QueueMessageHandlerFactory queueMessageHandlerFactory = new QueueMessageHandlerFactory();
    queueMessageHandlerFactory.setAmazonSqs(amazonSQSAsync);
    QueueMessageHandler messageHandler = queueMessageHandlerFactory.createQueueMessageHandler();
    return messageHandler;
}

@Bean
public AmazonSQSAsync awsSqsAsync() {
    AmazonSQSAsyncClient amazonSQSAsyncClient  = new AmazonSQSAsyncClient(new DefaultAWSCredentialsProviderChain());
    amazonSQSAsyncClient.setRegion(Region.getRegion(Regions.fromName(region)));
    return new AmazonSQSBufferedAsyncClient(amazonSQSAsyncClient);
}

I couldn't find any right documentation to correctly set retries so that if the retries exceed the threshold, the message should go to a dead-letter queue

Solution

  • If I am not mistaken, setting the maximum retries, and associated DLQ is done on the Broker side, and not configurable as part of the listener.

    Then in your code, you will do something like:

    @SqsListener(value = "MainQueue", deletionPolicy = SqsMessageDeletionPolicy.NEVER)
public void receive(String message, @Header("SenderId") String senderId, Acknowledgment ack) throws IOException {

  ack.acknowledge();  

}

    @SqsListener(value = "DLQ-AssociatedWithMain")
public void receiveDlq(String message) throws IOException {

}

    If Message is NOT acknowledged, then it will be retried for a specific max period, then sent to DLQ.

    === Edited ===

    The below for LocalStack are suggestions (Never Tested), However LocalStack (Free Version) as of now is supposed to support the AWS CLI:

    As such, if you look at the AWS CLI: you use aws create-queue to create a Queue, and --attributes if you wanted to specify DLQ Information, though I believe you must also create the DLQ queue as well before referencing the RN.

      create-queue
--queue-name <value>
[--attributes <value>]
[--tags <value>]
[--cli-input-json <value>]
[--generate-cli-skeleton <value>]

    DLQ Attribute Details:

    The following attributes apply only to dead-letter queues:

    RedrivePolicy – The string that includes the parameters for the dead-letter queue functionality of the source queue as a JSON object. The parameters are as follows:
    deadLetterTargetArn – The Amazon Resource Name (ARN) of the dead-letter queue to which Amazon SQS moves messages after the value of maxReceiveCount is exceeded.
    maxReceiveCount – The number of times a message is delivered to the source queue before being moved to the dead-letter queue. When the ReceiveCount for a message exceeds the maxReceiveCount for a queue, Amazon SQS moves the message to the dead-letter-queue.
RedriveAllowPolicy – The string that includes the parameters for the permissions for the dead-letter queue redrive permission and which source queues can specify dead-letter queues as a JSON object. The parameters are as follows:
    redrivePermission – The permission type that defines which source queues can specify the current queue as the dead-letter queue. Valid values are:
        allowAll – (Default) Any source queues in this Amazon Web Services account in the same Region can specify this queue as the dead-letter queue.
        denyAll – No source queues can specify this queue as the dead-letter queue.
        byQueue – Only queues specified by the sourceQueueArns parameter can specify this queue as the dead-letter queue.
    sourceQueueArns – The Amazon Resource Names (ARN)s of the source queues that can specify this queue as the dead-letter queue and redrive messages. You can specify this parameter only when the redrivePermission parameter is set to byQueue . You can specify up to 10 source queue ARNs. To allow more than 10 source queues to specify dead-letter queues, set the redrivePermission parameter to allowAll .

    https://docs.aws.amazon.com/cli/latest/reference/sqs/create-queue.html

    In LocalStack SQS Documentation, they have an example of creating an SQS Queue:

    awslocal sqs create-queue --queue-name sample-queue
{
    "QueueUrl": "http://localhost:4566/000000000000/sample-queue"
}

    So just take this example, create your DLQ, then create your Queue with --attributes to point to the DLQ RN.

    https://docs.localstack.cloud/aws/sqs/

    Hope this helps guide you in the right direction,

    ==== Edited ===

    Create Queue and DLQ using LocalStack:

    Create DLQ First:

    aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name MyDLQ --region eu-west-1

    Response:

    {
    "QueueUrl": "http://localhost:4566/000000000000/MyDLQ"
}

    Create an attributes.json file with the contents below:

    {
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-west-1:000000000000:MyDLQ\",\"maxReceiveCount\":\"1000\"}",
  "MessageRetentionPeriod": "259200"
}

    Create your main queue pointing to aattributes.json file:

    aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name MyMainQueue --attributes file://attributes.json --region eu-west-1

    Response:

    {
    "QueueUrl": "http://localhost:4566/000000000000/MyMainQueue"
}