Celery: How to shutdown/timeout a celery worker child/process

Question

I am running a celery worker --concurrency=10. So there are 10 process running in parallele on my environment.

When a task is executed and returns, I would like my server to tell the (worker) process to kill itself. Then, because of the concurrency, it should be create a new one.

How can I do this? I am thinking about SIGNALS -> task_postrun

Edit:

After setting up the application correctly (with --max-task-per-child that you can find below), I encountered an issue where I got exited code 143 from child processes.

This is because the process is killed and the server do not handle it.

The solution is to revoke the process before just after the task run:

from celery.signals import task_postrun
from celery.task.control import revoke

@task_postrun
def setup_task_postrun(task_id, task, *args, **kwargs):
    revoke(task_id=task_id, terminate=True)

I am not sure this is a good practice, but it works

Answer 1

That is relatively odd requirement, but it is quite easy to do with Celery. All you have to do is to put the following into your Celery configuration:

worker_max_tasks_per_child=1