Is there a way to perform some operations before exiting the celery worker when a keyboard-interrupt (say SIGINT) is encountered? I have tried using signals module in my tasks like so:
signal.signal(signal.SIGINT, keyboard_interrupt_handler)
But it doesn't work the way I expect it to. The keyboard_interrupt_handler does get executed but since it takes a few seconds to execute, the process is terminated before it can finish executing.
I have also tried to temporarily block the SIGINT signal with signal.pthread_sigmask and running the handler in a thread (since they can't be killed?) as well but it faces the same problem.
Is there a way to achieve something like this?
I suggest you try to implement a handler for the worker-shutdown signal. In general, Celery signals are extremely useful and Celery users should get familiar with them, Celery events too.
Sample code:
app = Celery('my_tasks', include['my_tasks.tasks', 'their_tasks.tasks'])
@worker_shutdown.connect
def handle_worker_shutdown(**kwargs):
mymodule.handle_worker_shutdown(app, _LOGGER, **kwargs)