ECS EC2 stack cdk deployment failing after first deployment

Question

I have a cluster that has a capacity of 1 t2.micro EC2 instance configured. I'm running 2 services with a desired task capacity of 1 each. Each of these tasks run different code but take 0.5 vCPU and 300mb memory each. I'm using cdk to deploy via my pipeline

Here's the problem:

Initially I had only 1 of these services configured and thus 0.5 vCPU free in the instance. After adding the second service, the first deployment succeeded. Now whenever I run cdk deploy again (without changing anything in the stack) I get the following:

service `xyz` was unable to place a task because no container instance met all of its requirements. The closest matching container-instance `abc` has insufficient CPU units available.

I'm clearly missing some key information about ECS or the deployment. I'd appreciate any advice.

Answer 1

Your cluster has a single t2.micro instance. A t2.micro instance has 1 CPU. You have two tasks deployed on that instance, each reserving 0.5 CPU. That means the entire CPU is reserved by the running tasks.

The default ECS deployment configuration attempts to deploy the new instance of the task, and wait for it to be healthy, before stopping the old instance of the task. That means you need an extra 0.5 CPU available to deploy a new instance of your task, and you don't have that available in your cluster currently.

You have some options to fix this:

• Configure auto-scaling in your cluster's capacity provider, to add an extra instance to your cluster when needed.
• Configure your service deployment configuration to stop the current running task before starting the new one.
• Switch from EC2 deployments to Fargate which doesn't have this sort of problem.