AWS SQS FIFO in Python - deleted messages are available even after correct VisbilityTimeout

Question

I created an FIFO queue. I send 10 msgs, receive them and delete them. But they are still available in queue and received again. I receive 10 msgs initially, then 5 , then 2 and then 0. "delete_message" response was also 200.

Code :

import json
import os
import uuid

import boto3
from dotenv import load_dotenv

load_dotenv()
sqs = boto3.client("sqs", os.getenv("AWS_REGION"))
receive_queue = os.getenv("RECEIVE_QUEUE_URL")


def receive(attempt_id, max_num_messages):
    response = sqs.receive_message(
        QueueUrl=receive_queue,
        ReceiveRequestAttemptId=attempt_id,
        MaxNumberOfMessages=max_num_messages,
        VisibilityTimeout=100,
        WaitTimeSeconds=20,
    )
    if "Messages" not in response:
        return None, True
    messages = [message["Body"] for message in response["Messages"]]
    receipt_handles = [message["ReceiptHandle"] for message in response["Messages"]]
    print(f"{len(messages)} msgs received")
    for receipt_handle in receipt_handles:
        sqs.delete_message(QueueUrl=receive_queue, ReceiptHandle=receipt_handle)
        receipt_handles.remove(receipt_handle)
    print(f"{len(receipt_handles)} msgs deleted")
    return messages, False


def send_to_queue(queue_url, data, message_group_id):
    response = sqs.send_message(
        QueueUrl=queue_url,
        MessageBody=json.dumps(data),
        MessageGroupId=message_group_id,
    )
    return response


# 10 msgs created
for i in range(10):
    sent_response = send_to_queue(
        receive_queue, {"key": i}, message_group_id=os.getenv("MESSAGE_GROUP_ID")
    )

# 10 msgs received
receive(str(uuid.uuid4()), 10)


# Now still 5 msgs are "inflight" and received again

Queue configuration

queue

I checked prev SO answers which asked me to change non-zero Timeout which I have set.

Answer 1

The issue is with this for loop:

    for receipt_handle in receipt_handles:
        sqs.delete_message(QueueUrl=receive_queue, ReceiptHandle=receipt_handle)
        receipt_handles.remove(receipt_handle)

It is attempting to loop through the receipt_handles list but in each loop it is removing elements from the list. This is confusing Python.

For example it starts the loop with element 0. In the next loop, it is operating on element 1 but element 0 was removed from the list so element 1 is actually the original element 2. It is therefore only deleting every second element.

If you do not call remove(), then it will correctly delete all the messages. You can see this by adding a debugging statement that prints a message whenever a message is deleted.