How to prevent celery.backend_cleanup from executing in default queue

Question

I am using python + flask + SQS and I'm also using celery beat to execute some scheduled tasks.

Recently I went from having one single default "celery" queue to execute all my tasks to having dedicated queues/workers for each task. This includes tasks scheduled by celery beat which now all go to a queue named "scheduler".

Before dropping the "celery" queue, I monitored it to see if any tasks would wind up in that queue. To my surprise, they did.

Since I had no worker consuming from that queue, I could easily inspect the messages which piled up using the AWS console. What is saw was that all tasks were celery.backend_cleanup!!!

I cannot find out from the celery docs how do I prevent this celery.backend_cleanup from getting tossed into this default "celery" queue which I want to get rid of! And the docs on beat do not show an option to pass a queue name. So how do I do this?

This is how I am starting celery beat:

/venv/bin/celery -A backend.app.celery beat -l info --pidfile=

And this is how I am starting the worker

/venv/bin/celery -A backend.app.celery worker -l info -c 2 -Ofair -Q scheduler

Keep in mind, I don't want to stop backend_cleanup from executing, I just want it to go in whatever queue I specify.

Thanks ahead for the assistance!

Answer 1

You can override this in the beat task setup. You could also change the scheduled time to run here if you wanted to.

app.conf.beat_schedule = {
   'backend_cleanup': {
       'task': 'celery.backend_cleanup',
       'options': {'queue': <name>,
                   'exchange': <name>,
                   'routing_key': <name>}
   }
}