I'm using Celery with Amazon SQS and I'm seeing loads of these in my logs:
[2013-06-19 19:03:42,890: ERROR/MainProcess] Pool callback raised exception: KeyError('8a8fe51c-b69f-498e-95c1-f037e2f2d4a8',)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/billiard/pool.py", line 129, in safe_apply_callback
fun(*args)
File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 301, in on_accepted
self.acknowledge()
File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 439, in acknowledge
self.on_ack(logger, self.connection_errors)
File "/usr/local/lib/python2.7/dist-packages/kombu/transport/base.py", line 98, in ack_log_error
self.ack()
File "/usr/local/lib/python2.7/dist-packages/kombu/transport/base.py", line 93, in ack
self.channel.basic_ack(self.delivery_tag)
File "/usr/local/lib/python2.7/dist-packages/kombu/transport/SQS.py", line 252, in basic_ack
delivery_info = self.qos.get(delivery_tag).delivery_info
File "/usr/local/lib/python2.7/dist-packages/kombu/transport/virtual/__init__.py", line 127, in get
return self._delivered[delivery_tag]
KeyError: '8a8fe51c-b69f-498e-95c1-f037e2f2d4a8'
I tried to google this and seems like nobody really knows where it's coming from? Does anybody know whats going on? Some strange behaviour I noticed, the task is failing and then it retries the same task again and again. After a while, not only does it retry, but it doesn't even show errors in the logs (or any log statement for that matter). It just says it got a task and it will try it. I'm not sure what happened?
I just did a purge all using the following:
from celery.task.control import discard_all
discard_all()
and somehow the task is coming back running again! Does anybody have any information how this is even possible?
This was caused by a bug in Kombu's SQS backend and was fixed in https://github.com/celery/kombu/pull/431