I have a list of integers (integerList) that I'd like to pass into an SQS queue where each message into the queue is an integer from the list.
I can do this one message at a time with the send_message()
command, and the code for that is below.
import boto3
sqsResource = boto3.resource('sqs')
def write_sqs(integerList):
queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
for i in integerList:
response = queue.send_message(MessageBody=str(i),
MessageGroupId='TESTING')
However, I'd like to speed up the function and send the messages in batches. Currently, AWS SQS allows batching up to 10 messages at a time with the send_messages()
command, but I'm not sure how to build the Entries=
attribute for the batch send. I'm breaking down the integerList into smaller lists of 10 using chunks = [integerList[x:x+10] for x in range(0, len(integerList), 10)]
, but the next steps are unclear.
Building on the answer from @liorko and some trial and error, this seems to work and is much faster than the 1-by-1 method I was using before.
import boto3
sqsResource = boto3.resource('sqs')
def write_sqs(integerList):
queue = sqsResource.get_queue_by_name(QueueName=NAMEOFQUEUEHERE)
maxBatchSize = 10 #current maximum allowed
chunks = [integerList[x:x+maxBatchSize] for x in range(0, len(integerList), maxBatchSize)]
for chunk in chunks:
entries = []
for x in chunk:
entry = {'Id': str(x),
'MessageBody': str(x),
'MessageGroupId': 'ANYTHINGYOUWANT'}
entries.append(entry)
response = queue.send_messages(Entries=entries)