New instances not getting added to ECS with EC2 deployment

Question

I am deploying a queue processing ECS service using EC2 deployment using CDK. Here is my stack

from aws_cdk import core, aws_ecs_patterns, aws_ec2, aws_ecs


class EcsTestStack(core.Stack):

def __init__(self, scope: core.Construct, construct_id: str, **kwargs) -> None:
    super().__init__(scope, construct_id, **kwargs)

    _container_image = aws_ecs.ContainerImage.from_asset(
        directory=".",
        file='Dockerfile.ECS_Test',
        exclude=["cdk.out"]
    )

    _scaling_steps = [{"upper": 0, "change": -1}, {"lower": 50, "change": +1}, {"lower": 100, "change": +2}]

    _vpc = aws_ec2.Vpc(self, "ecs-test-vpc-v4", max_azs=3)

    _cluster = aws_ecs.Cluster(self, "ecs-test-cluster-v4", vpc=_vpc)

    _cluster.add_capacity("ecs-autoscaling-capacity-v4",
                          instance_type=aws_ec2.InstanceType("t2.small"),
                          min_capacity=1,
                          max_capacity=3)

    self.ecs_test = aws_ecs_patterns.QueueProcessingEc2Service(
        self,
        "ECS_Test_Pattern_v4",
        cluster=_cluster,
        cpu=512,
        scaling_steps=_scaling_steps,
        memory_limit_mib=256,
        image=_container_image,
        min_scaling_capacity=1,
        max_scaling_capacity=5,
    )

The stack starts out with 1 task in the service and 1 EC2 instance. Based on my _scaling_steps, a new task should be added to the service when the number of messages in the queue > 50 and 2 new tasks should be added to the service. The stack starts out with 1 task in the service and 1 EC2 instance.

But when I add 200 new messages to the queue, I can see 1 new task added to my service and then I get this error message in the even.

service EcsTestStackV4-ECSTestPatternv4QueueProcessingService5C84D200-c00N6R56hB0p was unable to place a task because no container instance met all of its requirements. The closest matching container-instance 81e5380c718c4b558567dc6cc1fb1926 has insufficient CPU units available.

I also notice that no new EC2 instances were added.

Question: how do I get more EC2 instances added to my cluster when the service scales up?

Answer 1

I can see 1 new task added to my service and then I get this error message in the even.

This is because t2.small has 1000 CPU units. So your two tasks take all of them, and there is no other instances to place your extra task on.

I also notice that no new EC2 instances were added.

You set min_capacity=1 so you have only instance. The _scaling_steps are for the tasks only, not for your instances in autoscaling group. If you want more instance you have to set min_capacity=2 or whatever value you want.

I guess you thought that QueueProcessingEc2Service scales both instances and tasks. Sadly this is not the case.