Assign a specific number of workers for each Celery task in Python

Question

I have three Celery tasks:

• prediction
• training
• healthcheck

And I have 4 cpus in the machine. I would like to allocate 3 workers for prediction and training, and 1 for healthcheck. What is the easiest way to implement that? Note that I am going to schedule these tasks, and thus, I can't specify the number of workers in the apply_async() function directly.

Actual configs:

CELERY = Celery(
    CELERY_APP_NAME,
    backend=CELERY_BACKEND,
    broker=CELERY_BROKER,
    include=["src.tasks"],
)

CELERY.conf.update({"task_routes": {"src.tasks.*": {"queue": "input_queue"}},
                    }
                   )


@CELERY.task
def prediction():
    pass

@CELERY.task
def training():
    pass

@CELERY.task
def healthcheck():
    pass

And the command to run the worker:

celery --loglevel=INFO -A src.tasks worker -Q input_queue

Answer 1

This is how I would do it, because I could never truly understand celery multi:

• celery -A yourproject.yourapp -l info -c 3 -Q prediction
• celery -A yourproject.yourapp -l info -c 1 -Q healthcheck

After you run something similar to the above (with correct application parameter) you will end up with two workers subscribed to different queues. Your prediction and training tasks would be triggered with queue=prediction among other named parameters, while healthcheck task should be sent to the healthcheck queue in the similar way.