Search code examples
pythoncelerypython-multiprocessinggevent

How I can choose right type of worker in Celery?

I read documentation of Celery. I understood that concurrency can be implemented by different types: https://github.com/celery/celery/tree/master/celery/concurrency

How I can choose the right one (ex. Gevent of multiprocessing or eventlet) and why?

And another question - can I determine workers in .py files? Or I can only start workers from command line?

I have two types of tasks: 1.working with SQL databases and export data to csv files. 2.execute some sql on databases

Solution

  • Expanding on what @dejanlekic said:

    1. Choose multiprocessing / prefork as the default and only deviate from it if you have a particular reason.

    2. In production or production-like environments you should start celery workers from the command line using celery multi and a process manager of your choice like systemd or supervisord.