So as far as I can tell, if using any of the normal scaling methods like step scaling for Fargate the result when scaling down is that a random task is selected. However I'd like to scale down the task with the lowest cpu usage.
My use case is that each task is working on processing jobs, when a task finishes all processing jobs it sits idle. At that point I'd like to take that one down. Other tasks will still be working on jobs, I don't want to kill those tasks half way through and make them lose their progress.
My current solution is to have each task report their cpu usage to cloudwatch through cron, then have a lambda task to scale down any tasks which have had low cpu for multiple data points. But this feels like it may be overkill for a seemingly simple problem.
Just adding an update. A year on and we're still using a solution similar to what I described in the final paragraph of my original post. We have a database table listing how many tasks we want and what we want each of them working on, our central master service updates that as needed. These numbers are sent from our DB to cloudwatch, lambda reads from cloudwatch and either spins up new tasks as needed or kills the tasks with the lowest CPU if the task count is too high. It's very hacky but it works, and since there's no official way to do this with fargate the only other alternative would be to switch to something like K8s