Search code examples
celery

Celery worker setup and teardown

My Celery tasks require resources that are a bit expensive to setup but can be reused for many tasks on the same thread, but need to be closed when the worker shutdown. (Specifically, they require a fake X display using Xvfb used to render).

I can easily use a threading.local() variable and only initialize the resource once per thread, but the shutdown process is not so clear. The questions I have are:

  • Does Celery restart threads in the pool ever? If so, the resource needs to be restarted as well (or at least migrated)
  • Is there a way to detect when an entire worker is going down and shutdown the processes then?
Solution

  • You have signals for almost everything in Celery, so in your case, it sounds like you're looking for worker-process-shutdown signal.

    I guess you can do something like (pseudo code):

    @worker_process_shutdown.connect
def shutdown(*args, **kwargs):
    # release your resources here..
    pass