I'm trying to setup a SQS broker with a celery app, configured in a django project. Here's my setup:
celery.py:
import os
from celery import Celery
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "core.settings")
app = Celery("my-app")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()
My celery related django settings:
CELERY_BROKER_TRANSPORT_OPTIONS = {
"region": "eu-west-1",
"queue_name_prefix": f"celery-",
}
CELERY_BROKER_URL = "sqs://"
CELERY_TASK_DEFAULT_QUEUE = "default"
This configuration works fine using a rabbitmq broker, which makes me think the overall configuration is correct, but when I use SQS as broker, the messages are sent to SQS (the "Message available" counter increases), but once picked up by my worker, they go to "Messages in flight" and stay there forever it seems (at least for hours). Also, I can see the worker logs doing things, but never actually executes the task for some reason.
Here are some celery worker logs:
host;x-amz-date
786cb490d758593ebf5a6e0c0b34cf025b3309bb0d777891344fd32bd01cb61b
[2023-11-19 19:52:05,766: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20231119T195205Z
20231119/eu-west-1/sqs/aws4_request
32a2b642b24ffcd3f42216826611c27799efb1cf756b26e7119d5553009f527c
[2023-11-19 19:52:05,766: DEBUG/MainProcess] Signature:
93bcf47703548603fa0787fbbd2ae8aa4b54460db91695f4f6c818b9141b620e
[2023-11-19 19:52:10,959: DEBUG/MainProcess] Response headers: {}
[2023-11-19 19:52:10,960: DEBUG/MainProcess] Response body:
b'<?xml version="1.0"?><ReceiveMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ReceiveMessageResult><Message><MessageId>c4e1b404-.....-c046dc4aeb60</MessageId><ReceiptHandle>AQEBy+4G0x9BF8su10zFWJQuyEXJOFSxF........cRPUdk/vjWzirMiBw97ZSG44M=</ReceiptHandle><MD5OfBody>483202aa2938b...da1b024b63</MD5OfBody><Body>eyJib2R5IjogIlcxdGRMQ0I3...CJyZXBseV90byI6ICJmOWZjY2FmDIyMTBkYTAyMiJ9fQ==</Body><Attribute><Name>ApproximateReceiveCount</Name><Value>1</Value></Attribute></Message></ReceiveMessageResult><ResponseMetadata><RequestId>28a76707-9a01...d3f3a42ad7</RequestId></ResponseMetadata></ReceiveMessageResponse>'
[2023-11-19 19:52:10,960: DEBUG/MainProcess] Event choose-signer.sqs.ReceiveMessage: calling handler <function set_operation_specific_signer at 0xffff9b901750>
[2023-11-19 19:52:10,960: DEBUG/MainProcess] Calculating signature using v4 auth.
[2023-11-19 19:52:10,960: DEBUG/MainProcess] CanonicalRequest:
POST
/ACCOUNT_ID/celery-default
host:sqs.eu-west-1.amazonaws.com
x-amz-date:20231119T195210Z
host;x-amz-date
786cb490d758593ebf5a6e...d777891344fd32bd01cb61b
[2023-11-19 19:52:10,961: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20231119T195210Z
20231119/eu-west-1/sqs/aws4_request
a6b5c2bb4cb9eef89...4e16bc164d0a0d9b1a1
[2023-11-19 19:52:10,961: DEBUG/MainProcess] Signature:
81bd2621f3ce88b5a76...79dc18951c3bc0c8a
[2023-11-19 19:52:21,000: DEBUG/MainProcess] Response headers: {}
[2023-11-19 19:52:21,000: DEBUG/MainProcess] Response body:
b'<?xml version="1.0"?><ReceiveMessageResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><ReceiveMessageResult/><ResponseMetadata><RequestId>802bb8a8-...-e57c9a538959</RequestId></ResponseMetadata></ReceiveMessageResponse>'
[2023-11-19 19:52:21,000: DEBUG/MainProcess] Event choose-signer.sqs.ReceiveMessage: calling handler <function set_operation_specific_signer at 0xffff9b901750>
[2023-11-19 19:52:21,001: DEBUG/MainProcess] Calculating signature using v4 auth.
[2023-11-19 19:52:21,001: DEBUG/MainProcess] CanonicalRequest:
POST
/ACCOUNT_ID/celery-default
I don't think it's a permission issue, as I can reproduce the error either on AWS (fargate) using an execution role, or locally using an admin user, with same result in both cases.
Do you have any idea what's wrong please?
In the end, updating celery (and kombu) fixed the issue!