Search code examples
amazon-web-servicesamazon-sqs

When should I delete messages in SQS?

My application consists of:

  • 1 Amazon SQS message queue
  • n workers

The workers have the following logic: 

 1. Wait for message from SQS queue
 2. Perform task described in message
 3. Delete the message from the SQS queue
 4. Go to (1)

I want each message to be received by only one worker to avoid redundant work.

Is there a mechanism to mark a message as "in progress" using SQS, so that other pollers do not receive it?

Alternatively, is it appropriate to delete the message as soon as it is received? 

 1. Wait for message from SQS queue
 2. Delete the message from the SQS queue
 3. Perform task described in message
 4. Go to (1)

If I follow this approach, is there a way to recover received but unprocessed messages in case a worker crashes (step (3) fails)?

This question is specific to Spring, which contains all sorts of magic.

Solution

  • An SQS message is considered to be "inflight" after it is received from a queue by a consumer, but not yet deleted from the queue. These messages are not visible to other consumers.

    In SQS messaging, a message is considered in "inflight" if:

    1. You the consumer have received it, and
    2. the visibility timeout has not expired and
    3. you have not deleted it.

    SQS is designed so that you can call ReceiveMessage and a message is given to you for processing. You have some amount of time (the visibility timeout) to perform the processing on this message. During this "visibility" timeout, if you call ReceiveMessage again, no worker will be returned the message you are currently working with. It is hidden.

    Once the visibility timeout expires the message will be able to be returned to future ReceiveMessage calls. This could happen if the consumer fails in some way. If the process is successful, then you can delete the message.

    The number of messages that are hidden from ReceiveMessage call is the "inflight" number. Currently a SQS queue is set by default to allow a max of 120,000 messages to be "inflight".

    http://docs.amazonwebservices.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html