Search code examples
amazon-web-servicesamazon-ecsaws-auto-scaling

ECS Agent cannot start! Unable to initialize Task ENI dependencies


I encountered a problem while starting an instance from an Autoscale Group: the ecs-agent container starts, but then encounters an error and stops. This is the error:

Unable to initialize Task ENI dependencies: unable to create ENI watcher: unable to create udev monitor: permission denied

The group is connected to a ECS cluster via a capacity provider and everything is working in the same VPC. Now, the launch template, autoscaling group and tasks were all working before I hade to make some changes to the ECR image, create a Lambda, stopped the instance and tried to restart it again.

Here is some info from the launch template:

  • AMI ID : ami-0e174c3e9fa61e692

  • Instance type : g4dn.2xlarge

  • IAM profile : ecsInstanceRole

  • User Data :

    #!/bin/bash
    cd ~
    curl -O https://s3.eu-west-1.amazonaws.com/amazon-ecs-agent-eu-west-1/amazon-ecs-init-latest.amd64.deb
    sudo dpkg -i amazon-ecs-init-latest.amd64.deb
    sudo mkdir /etc/ecs/
    sudo echo ECS_CLUSTER=*** >> /etc/ecs/ecs.config
    sudo echo ECS_ENABLE_GPU_SUPPORT=true >> /etc/ecs/ecs.config
    sudo systemctl start ecs
    

I tried to stop and start the instance manually and through changing the autoscaling capacities (max, min, desired). I rolled back every change I made before encountering the error and it still did not work.

I expect to be able to start the Instance via the autoscaling group, that the ecs-agent container could start and that the connected ECS Task can then be provided to the instance.

Any help is well accepted! Should you need more details I can give you some. Thanks and good day!


Solution

  • I had the exact same issue and found out that it happened due to an ECS agent update. It was working fine in version 1.77.0 and stopped working in 1.78.0. AWS reverted the bad commit, see https://github.com/aws/amazon-ecs-agent/pull/3993, and released 1.78.1 which worked fine here.