Search code examples
concurrencyceleryweb-workerdjango-celeryceleryd

Celery: number of workers vs concurrency

What is the difference between having:

  1. one worker with concurrency 4 or
  2. two workers with concurrency 2 each

for the same queue.

Thanks

Solution

  • Intersting question.

    Things that I can think of (I'm sure there are a lot more):

    • For having high availability:
      1. You want more than one machine (if one goes down) - so you must use worker per machine.
      2. Even for one machine - I think it is safer to have 2 workers which run in a two different processes instead of one worker with high concurrency (correct me if I wrong, but I think it is implemented with threads).
    • In docs I see that they the recommendation is to use concurrency per CPUs.
    • If you want to separate different tasks to different workers..

    Of course, you have price for that: more processes that takes more resources (CPU/Memory etc).

    I found this question which is quite similar.