Search code examples
djangorediscelerysystemddjango-celery

Wait for current task to complete before stopping celery

I have created a systemd process that runs celery. I have set the concurrency to 1 since I need to tasks to run on a first in first out(FIFO) queue.

If i need to stop celery for some maintenance, how do I make celery wait for the currently running task to complete before stopping? I don't want it to wait for the entire queue to complete. I just need celery to wait for the current task to complete running before celery itself stops.

This is my systemd service file

[Unit]
Description=celery daemon
After=network.target

[Service]
User=<username>
Group=<group>
WorkingDirectory=/path_to_django_project/project_name
ExecStart=celery -A project_name worker --concurrency=1

[Install]
WantedBy=multi-user.target
Solution

  • That is exactly what is happening. - When you initiate graceful shutdown, Celery worker will unsubscribe from the queue, wait for the tasks executed by its own worker processes to finish, and then it will shut down.

    Also, your ExecStart line is incorrect. The concurrency setting is either set with -c or --concurrency. Here is the quote from the worker help:

    -c CONCURRENCY, --concurrency CONCURRENCY Number of child processes processing the queue. The default is the number of CPUs available on your system.

    Concurrency equal to 1 seems wrong, unless the machine where your Celery worker is running is single-core.

    Daemonization section of the Celery documentation explains how to to use systemd, with an example.