EC2 cannot access S3 in the same account with proper IAM role

Question

I have two EC2 instances: ec2-1 and ec2-2, one s3 bucket: bucket-1, all in the same aws account.

both instances are associated with the same IAM role which has this trust relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

for permissions policy, this IAM role has AmazonS3FullAccess

I have set up aws credentials via aws configure, typed in the same access/secret keys on both instances.

These two instances have the same VPC, security group, but somehow, ec2-1 could access bucket-1 via aws cli, while ec2-2 cannot and threw this error:

aws s3 ls

An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

Could anyone help advise here?

Thanks!

Answer 1

Normally, permissions are granted to an Amazon EC2 instance by assigning an IAM Role to the instance. Any software on the instance that uses an AWS SDK will automatically retrieve credentials associated with the IAM Role.

However, you then mention that you used aws configure to store credentials on the instance. This will likely override the permissions granted via the IAM Role.

You can use aws sts get-caller-identity to display which IAM entity is currently being used to call AWS. This will help you debug which entity is being used so you can then check if that entity has sufficient permissions.

If I had to guess, one instance is probably using the IAM Role permissions (and is working) while the other is using the locally-stored credentials (and is failing).