Search code examples
celerydjango-celeryworker

Celery : CELERYD_CONCURRENCY and number of workers

From the other stackoverflow answer, I've tried to limit celery's number of workers

After I terminated all the celery worker, I restarted celery with new configuration.

CELERYD_CONCURRENCY = 1 (in Django's settings.py)

Then I typed following command to check how many celery workers are working.

ps auxww | grep 'celery worker' | grep -v grep | awk '{print $2}'

It returns two PIDs like 24803, 24817.

Then I changed configuration to CELERYD_CONCURRENCY = 2 and restarted celery.

Same command returns three PIDs like 24944, 24958, 24959. (As you can see, last two PIDs are sequential)

It implies that number of workers is increased as I expected.

However, I don't know why it returns two PIDs even though there is only one celery worker is working?

Is there a something subsidiary process to help worker?

Solution

  • I believe one process always acts like the controller that listens for tasks and then distributes them to it's child processes to actually perform the work. Therefore, you will always have 1 more process than the configuration setting.