Search code examples
python-3.xcelerydjango-celeryflower

Celery task suddenly missing


I have a long running celery task which I can see it in flower at the beginning of execution but after some hours, it's like it doesn't exist! All I see in task details page is Unknown task '894a8b45-5963-40da-a104-7ffff98bc267' and I cannot find that key in Redis but I can tell the task is still running by looking at logs. The task does not fail, it just disappears!


Solution

  • flower keeps task's information in-memory. It is not related to your celery backend (Redis for example).

    By default it keeps the last 10,000 tasks so if your system run many tasks it make sense that your task becomes older and being purged to make place for a newer tasks. Assuming that's the case - you'll see the Unknown task message.

    You can tune the number of tasks to keep in memory with max-tasks.