Search code examples
amazon-web-servicesamazon-ec2amazon-ecs

AWS ECS placement memory issue


I have 3 ECS services:

  • Service512 - 512mb container memory
  • Service1280 - 1280mb container memory
  • Service768 - 768mb container memory

They are placed on 3 t3a.small instances (each instance has also 2 daemons which grab 640mb in sum)

Instances diagram

Regardless of placement strategy, next Service512 deployment will place it on instance 3 Instances diagram after deployment, so there will be no space to deploy Service1280. Is instance 4 required so there are enough space for Service1280 deployment, or are there any workaround to use resources more optimally?


Solution

  • So the problem is happening during deployments. This is actually entirely expected behavior, since ECS is not going to deregister and remove the old Service512 instance until the new instance is up and healthy. And it's going to want to do the same thing with the Service1280, which means you will need a 4th instance, although after the deployment the 2nd instance should be removed automatically by the capacity provider.

    You have to have enough EC2 instances to handle not only the desired service count, but also the desired healthy task percentage during a deployment. If you have the capacity provider configured with an EC2 auto-scaling group, it should automatically create the 4th instance needed for the Service1280 deployment, and then after the deployment it should automatically delete the 2nd instance in your diagram, leaving you back with 3 instances.

    You can change the deployment settings minimumHealthyPercent and maximumPercent values if you want ECS to delete the running task before deploying the new version, which would remove the need for a temporary 4th instance in your cluster, but of course that will incur down-time for anyone using your service.