I have a python app that is supposed to read all the messages from the queue, and then process them after all the messages have been added to a list. Once the process is complete it deletes the messages in batch.
The app works perfectly for up to 100K messages however while doing a load test for more than 500K messages.
I got the following error
botocore.errorfactory.OverLimit: An error occurred (OverLimit) when calling the ReceiveMessage operation:
Too many messages have been received without being deleted
I understand that this is because I am trying to fetch a large number of messages without deleting them, What I want to know is if there is a way to bypass this.
Quota limit is around 120K. You can increase the quota with a support request.
For most standard queues (depending on queue traffic and message backlog), there can be a maximum of approximately 120,000 inflight messages (received from a queue by a consumer, but not yet deleted from the queue). If you reach this quota while using short polling, Amazon SQS returns the OverLimit error message. If you use long polling, Amazon SQS returns no error messages. To avoid reaching the quota, you should delete messages from the queue after they're processed. You can also increase the number of queues you use to process your messages. To request a quota increase, submit a support request. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/quotas-queues.html