Why can I not access AWS Secrets Manager from ECS when instance role has permissions?

Question

I have an ECS task definition that points at an image I am running. The task definition's Task Execution role includes the following permissions...

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": "arn:aws:secretsmanager:us-east-2:...:secret:dev/CBusHA/Backend-qoy4yx"
        }
    ]
}

But when I try to run the following from the instance...

    public String getSecret(String secretName){
        SecretsManagerClient secretsManager = SecretsManagerClient.builder()
                .region(Region.US_WEST_2) // Change to your desired region
                .build();

        GetSecretValueRequest getSecretValueRequest = GetSecretValueRequest.builder()
                .secretId(secretName)
                .build();

        GetSecretValueResponse getSecretValueResponse = secretsManager.getSecretValue(getSecretValueRequest);

        secretsManager.close();

        return getSecretValueResponse.secretString();
    }

I get

software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profilesAndSectionsMap=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profilesAndSectionsMap=[])): Profile file contained no credentials for profile 'default': ProfileFile(profilesAndSectionsMap=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]

I also made sure Secrets Manager had the following resource Permission

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::...:role/ecsTaskExecutionRole"
      },
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*"
    }
  ]
}

What am I missing? How do I add the permissions to allow access to SecretsManager from ECS?

Answer 1

Instead of the task execution role, you need to use a Task Role.

• Task role: If your containerized applications need to call AWS APIs, they must sign their AWS API requests with AWS credentials, and a task IAM role provides a strategy for managing credentials for your applications to use, similar to the way that an Amazon EC2 instance profile provides credentials to Amazon EC2 instances.

• Task execution role: Grant the ECS agents permission to make AWS API calls who are responsible for managing the tasks in the cluster.

In your task definition, specify the taskRoleArn property.

{
"taskDefinition": {
    "memory": "128", 
    .
    .
    . 
    "taskRoleArn": "arn:aws:iam::%ACCOUNT_ID%:role/SOME-ROLE", 
    "taskDefinitionArn": "arn:aws:ecs:%REGION%:%ACCOUNT_ID%:task-definition/my-task-name:%BUILD_NUMBER%",
    .
    .
    .
    }
}