Why is a celery task not consumed by a worker that is free?
I have an application in Python with a microservices architecture and pipelines for which I use Celery along with RabbitMQ and Redis. In an application flow (machine learning training) 8 methodologies are needed, therefore a first worker called "Worker Training" sends 8 tasks to another worker called "Worker Training Model". This second one has 3 replicas to be able to finish the training faster. At first it works well, each worker consumes a methodology and processes it until finishing and consuming the next one. However I am seeing that for example at this moment, 5 of the 8 tasks have already been completed, there are 2 workers (of the 3 replicas) processing a task, but there is 1 worker doing nothing! It should be processing the last missing methodology! Any idea why this happens? I think that in the end it will end up being processed in one of the other 2 workers when they finish with the one they have at the moment, but I need to be more efficient and not have workers doing nothing that can consume tasks. My RabbitMQ dashboard looks like this right now:
(That task is the missing methodology and should be done by the free worker...)
This is most likely caused by the prefetching. Read the Prefetch Limits section, and the section after to find out how to make worker reserve as many tasks as they have (free) worker processes.
In short (TLDR):
task_acks_late = True
worker_prefetch_multiplier = 1