AWS SQS / Lambda -> Process Data 1 Record At a Time with Delay Between Message Sent or Synchronously

Question

I am looking for a solution that does the following using SQS (or any other AWS service really):

Use Case Two: If I have X (1-10 messages per second) amount of messages being sent to an SQS queue from a lambda function. I would like to send messages 1-by-1 to another lambda function from the SQS Queue synchronously. For example, my lambda function sends messages to the SQS queue; it sends 1 message every 0.1 seconds. So 10 messages every second. I would like my SQS queue to send data 1 message at a time to my destination function for processing, and only send the next message when the previous message has finished processing. So in this case, message one is processed straight away, the second message sits in the queue till the first message is complete, once complete the second message get sent, so so and so on.

Use Case Two: If I have X (1-10 messages per second) amount of messages being sent to an SQS queue from a lambda function. I would like to send messages 1-by-1 to another lambda function from the SQS Queue with a delay between each message sent. For example, message one is sent to my destination, 3 seconds later message two gets sent, three seconds later message three gets sent, etc. I choose 3 seconds here as that is the maximum time it takes for my destination process to run.

TL;DR I need a way to only send data from a queue one record at a time; with either a delay of 3 seconds between each message or until the previous message has finished processing.

What I have done so far?

I thought creating a FIFO queue with a 3-second delay would do the trick here?

  GSSAREventQueue:
    Type: AWS::SQS::Queue
    Properties: 
      ContentBasedDeduplication: false
      DeduplicationScope: messageGroup
      FifoThroughputLimit: perMessageGroupId 
      DelaySeconds: 3
      FifoQueue: true
      ReceiveMessageWaitTimeSeconds: 3
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt MyDeadLetterQueue.Arn
        maxReceiveCount: 10

  MyDeadLetterQueue: 
    Type: AWS::SQS::Queue
    Properties: 
      FifoQueue: true
 

But when I look in the logs of my lambda which the data is being sent to:

2021-09-30T14:49:28.816+01:00   END RequestId: 5f9e784c-bba1-56f5-84e7-2183793ac607

2021-09-30T14:49:28.816+01:00   REPORT RequestId: 5f9e784c-bba1-56f5-84e7-2183793ac607 Duration: 217.39 ms Billed Duration: 218 ms Memory Size: 128 MB Max Memory Used: 101 MB XRAY TraceId: 1-6155c068-6565d6de237fd0170fb290d0 SegmentId: 11f9660438a56763 Sampled: true

2021-09-30T14:49:30.808+01:00 START RequestId: 7d98c954-47c6-5b7f-8665-909b855be17b Version: $LATEST

there is literally no delay. I made sure to send the data in batches of 1 on my Lambda

      Events:
        UserEvent:
          Type: SQS
          Properties:
            BatchSize: 1
            Queue: !GetAtt GSSAREventQueue.Arn

I am likely doing something wrong here; is my use case possible?

What I thought could be happening:

SQS is sending data with a 3-second delay between each batch of messages? Maybe it's sending 10 messages at a time, and because I have set the BatchSize: 1 on the lambda, it is sending data through 1 at a time?

--

The main reason for needing this is I am reading/writing data from the same record in the DynamoDB Table; and if the lambda gets 10 messages at the same time and 10 lambdas are processing separately, the numbers are going to be wrong.

Answer 1

You may use the reserved concurrency feature of lambda. https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html

You may set the value of reserved concurrency to 1 and batch size to 1 for the destination lambda. This will ensure that only one instance of destination lambda will run at a time and will pick up one message at a time from SQS.