I have an existing application that is using Celery.
Clients submit tasks to Celery, and Celery's workers unstack those tasks and run it, accross different physical hosts.
Each Celery worker run a task at once. A given physical host has multiples workers.
Our tasks may have different ressources requirements. A task may need (1CPU, and 5GB RAM), and another (1CPU, and 20GB RAM) to work, and so on.
The problem is that Celery is just backed by a simple producer-consumer system, and at a given time, I have situation where some tasks terminate with Out Of Memory error. That's because the total RAM used by workers tasks are superior to the physical host total memory. Indeed, workers unstack tasks without any particular intelligence regarding ressources consumed and available.
A solution might be to set a memory limit on what task every worker is running. The problem is that workers are initialized statically, and my tasks have no static requirements. So i'm afraid it will endup in a resource waste.
I have used lot of scheduler from the past (YARN, PBS, MESOS, ..), and it is convenient for dynamic workload (dynamic tasks requirement), because you can define the CPU/RAM requirements for each task you submit. And it's the role of the scheduler to ensure that its dynamically allocated containers do not overcommit the physical memory.
So two possibilies : either Celery is just a task queue and IS not a real scheduler, so it is the wrong tool for doing what I want, either I miss some useful Celery options to do what I want ?
Thank you very much
Celery does not do what you want, the way you want.
It does not, however, mean that you can't do it with Celery. If your taskX needs 4cores and 12G of RAM, what you can do (and indeed that is how we do it) is to have a queue called "taskx_q" and have servers (AWS EC2 instances with 4 cores and ~12G RAM) running celery as service (systemd) subscribed to this particular queue, with concurrency set to 1. Yes, it requires some coding to implement good autoscaling (autoscaling groups can help), but it is nothing difficult.