Search code examples
pythonaws-lambdaboto3amazon-sqs

How to send multiple batch messages from a list to an SQS queue using Boto3 and Python


I have a list of integers (integerList) that I'd like to pass into an SQS queue where each message into the queue is an integer from the list.

I can do this one message at a time with the send_message() command, and the code for that is below.

import boto3

sqsResource = boto3.resource('sqs')

def write_sqs(integerList):
    queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
    for i in integerList:
        response = queue.send_message(MessageBody=str(i),
                                      MessageGroupId='TESTING')

However, I'd like to speed up the function and send the messages in batches. Currently, AWS SQS allows batching up to 10 messages at a time with the send_messages() command, but I'm not sure how to build the Entries= attribute for the batch send. I'm breaking down the integerList into smaller lists of 10 using chunks = [integerList[x:x+10] for x in range(0, len(integerList), 10)], but the next steps are unclear.


Solution

  • Building on the answer from @liorko and some trial and error, this seems to work and is much faster than the 1-by-1 method I was using before.

    import boto3
    
    sqsResource = boto3.resource('sqs')
    
    def write_sqs(integerList):
        queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
        maxBatchSize = 10 #current maximum allowed
        chunks = [integerList[x:x+maxBatchSize] for x in range(0, len(integerList), maxBatchSize)]
        for chunk in chunks:
            entries = []
            for x in chunk:
                entry = {'Id': str(x), 
                         'MessageBody': str(x), 
                         'MessageGroupId': 'ANYTHINGYOUWANT'}
                entries.append(entry)
            response = queue.send_messages(Entries=entries)