Search code examples
pythoncelery

How are Celery queues managed?


This is a bit of theoretical question and I cant find a clean cut answer in any celery documentation I ve read. So assume we have a Flask app and also Celery with either redis or rabbitmq. Now the broker here might be important but please let me know how it affects this if you can.

So our Flask app send some celery tasks with mytask.apply_async((args, args), time_limit=5). On the other side I have some workers subscribing to a queue task_routes = {'tasks.mytask': {'queue': 'transformations'}} I ve read in the Celery docs that the queue is created when a celery worker starts and subscribes to it. So there are a few things here which I can find answers in documentation:

  • Is that correct the queues disappear if the workers shut down? What if I want the queue to exist even if no workers are alive so when the worker start up again they can continue processing messages that were received when the were inactive. Can this be achieved?
  • Who manages the routing to the celery queues? The broker receives my task, but my sender does not know anything about the queues. The queues, based on documentation, are defined in the worker side. How is the task routed to the correct queue? Does the queue has to be somehow defined both on the sender and the worker side?

Solution

  • Ok I found some usefull resources:

    • Here an article about how queues, exchanges and workers are coordinated with Python and RabbitMQ. Basically, to answer the second question the queues are created by the workers and the broker(RabbitMQ) does the routing
    • Here some info from RabbitMQ about the lifecycle of the queues. So to asnwer the first question a queue can persist OR be auto-deleted if all workers die. So the user can configure it either way