Cloudformation template for creating ECS service stuck in CREATE_IN_PROGRESS

Question

I am creating an AWS ECS service using Cloudformation.

Everything seems to complete successfully, I can see the instance being attached to the load-balancer, the load-balancer is declaring the instance as being healthy, and if I hit the load-balancer I am successfully taken to my running container.

Looking at the ECS control panel, I can see that the service has stabilised, and that everything is looking OK. I can also see that the container is stable, and is not being terminated/re-created.

However, the Cloudformation template never completes, it is stuck in CREATE_IN_PROGRESS until about 30-60 minutes later, when it rolls back claiming that the service did not stabilise. Looking at CloudTrail, I can see a number of RegisterInstancesWithLoadBalancer instantiated by ecs-service-scheduler, all with the same parameters i.e. same instance id and load-balancer. I am using standard IAM roles and permissions for ECS, so it should not be a permissions issue.

Anyone had a similar issue?

Answer 1

Your AWS::ECS::Service needs to register the full ARN for the TaskDefinition (Source: See the answer from ChrisB@AWS on the AWS forums). The key thing is to set your TaskDefinition with the full ARN, including revision. If you skip the revision (:123 in the example below), the latest revision is used, but CloudFormation still goes out to lunch with "CREATE_IN_PROGRESS" for about an hour before failing. Here's one way to do that:

"MyService": {
    "Type": "AWS::ECS::Service",
    "Properties": {
        "Cluster": { "Ref": "ECSClusterArn" },
        "DesiredCount": 1,
        "LoadBalancers": [
            {
                "ContainerName": "myContainer",
                "ContainerPort": "80",
                "LoadBalancerName": "MyELBName"
            }
        ],
        "Role": { "Ref": "EcsElbServiceRoleArn" },
        "TaskDefinition": {
            "Fn::Join": ["", ["arn:aws:ecs:", { "Ref": "AWS::Region" },
            ":", { "Ref": "AWS::AccountId" },
            ":task-definition/my-task-definition-name:123"]]}
        }
    }
}

Here's a nifty way to grab the latest revision of MyTaskDefinition via the aws cli and jq:

aws ecs list-task-definitions --family-prefix MyTaskDefinition | jq --raw-output .taskDefinitionArns[0][-1:]