Celery - Check if worker received SIGTERM

Question

I have a Celery task that is quite long. More than a few minutes.

Sometimes, for various reasons, a worker is marked for termination and another worker starts. This can happen if the machine it runs on needs to be replaced, or a new code version is being deployed. In that case, the worker receives the SIGTERM signal.

I'm wondering if it's possible for the task itself to periodically check whether this worker has received SIGTERM and is pending termination, and in that case just put the task back in the queue and terminate. (The task will then be started on another worker, and will continue doing its work)

EDIT: Clarifying - Is it possible within the task to check whether it is executed on a worker that waits to be terminated. Like this:

# Some long task that can take even a few hours.
def some_task(...):
    for i in range(...):
        do_some_work()
        # That's the missing function:
        if did_this_worker_received_SIGTERM_and_waiting_to_be_terminated():
             # stop the task in the middle, and it will be executed again later

Answer 1

When Celery worker receives SIGTERM, it will initiate the warm shutdown. It means it will unsubscribe itself from all the queues, prefetched tasks (if any) will go back to their queues, and worker itself will start waiting for the currently running tasks to finish before it shuts down. No task is lost, if that is what you fear.

All these events can be handled (see Worker Signals).

If you still insist on having some extra logic inside your tasks that handles worker state, then perhaps the easiest solution is to implement worker shutdown handler (as described in the doc section I mentioned above), make it store a flag in Redis or some other distributed K/V storage), and refactor your tasks that need this so they access this flag and perform whatever action you need them to perform.