Amazon ECS - Permission denied when using IAM role on Docker entrypoint

Question

I'm looking for a way to inject secrets/certificates into Amazon ECS containers. In my case, it's a simple nginx container.

I've been following this post, using AWS Parameter Store: https://aws.amazon.com/blogs/compute/managing-secrets-for-amazon-ecs-applications-using-parameter-store-and-iam-roles-for-tasks/

Here's the basic gist:

1. On my Dockerfile, I attach a script on entrypoint which installs the AWS client and fetches the keys from AWS parameter store.

Dockerfile

FROM nginx:1.16.0

...
ENTRYPOINT ["/var/run/fetch.sh", "nginx", "-g", "daemon off;"]

fetch.sh

        aws ssm get-parameter \
            --name ${key} \
            --with-decryption \
            --region us-east-1 \
            --output text \
            --query Parameter.Value

2. The task definition assumes an IAM role that has access to the required services (kms + parameter store). I can verify this works because if I ssh to the server and run the script on the container, I am able to fetch the keys from Parameter Store.

  {
    "portMappings": [
      {
        "hostPort": 0,
        "protocol": "tcp",
        "containerPort": 443
      }
    ],
    "cpu": 0,
    "environment": [],
    "mountPoints": [],
    "memoryReservation": 256,
    "memory": 512,
    "volumesFrom": [],
    "image": "url/some_image:latest",
    "essential": true,
    "name": "my-container"
  }

3. When ECS runs this task, it should hit the entrypoint which fetches the keys from parameter store and saves them.

I'm able to fetch the keys on a running task by running it manually via docker exec, but I'm unable to fetch them when starting a task (specifically when I attach the script on the entrypoint as on code above).

Does an ECS task have access to IAM roles at the entrypoint? When does it actually assume IAM roles?

Answer 1

You can now easily inject secrets from SSM or Secrets Manager using the secrets in the containerDefinitions of a task definition. With this solution, you don't have to run/manage your custom scripts to fetch your secrets anymore.

It looks like this:

{
    "containerDefinitions": [{
        "secrets": [{
            "name": "environment_variable_name",
            "valueFrom": "arn:aws:secretsmanager:region:aws_account_id:secret:secret_name-AbCdEf"
        }]
    }]
}

{
    "containerDefinitions": [{
        "secrets": [{
            "name": "environment_variable_name",
            "valueFrom": "arn:aws:ssm:region:aws_account_id:parameter/parameter_name"
        }]
    }]
}

Have a look at AWS Launches Secrets Support for Amazon Elastic Container Service and Specifying Sensitive Data.

You must have a task execution role and reference it in your task definition. Example policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameters",
        "secretsmanager:GetSecretValue",
        "kms:Decrypt"
      ],
      "Resource": [
        "arn:aws:ssm:<region>:<aws_account_id>:parameter/parameter_name",
        "arn:aws:secretsmanager:<region>:<aws_account_id>:secret:secret_name",
        "arn:aws:kms:<region>:<aws_account_id>:key/key_id"
      ]
    }
  ]
}

More info in Required IAM Permissions for Amazon ECS Secrets.