Why is CloudFormation using a strange role to execute updates?

Question

I was trying to update a stack to add a policy to a role. The role I used to push the stack updates had AdministratorAccess, so I expected the updates to not run into permission problems. However, I got a strange error message:

API: iam:PutRolePolicy User: arn:aws:sts::xxxxx:assumed-role/NotAdmin/yyyy is not authorized to perform: iam:PutRolePolicy on resource: role zzzz because no identity-based policy allows the iam:PutRolePolicy action

The ARN of the role in the error message is a different user from the one I used to push the updates. It was a role with limited permissions that a script uses to periodically update that stack.

I tried using an explicit --role-arn argument:

aws cloudformation update-stack --region xxx --stack-name xxx --role-arn arn:aws:iam::xxx:role/RoleWithAdminAccess

but then I received this error message:

An error occurred (ValidationError) when calling the UpdateStack operation: Role arn:aws:iam::xxx:role/RoleWithAdminAccess is invalid or cannot be assumed

How can I get this stack update to work?

Answer 1

This issue is happening because --role-arn is "sticky"

From https://docs.aws.amazon.com/cli/latest/reference/cloudformation/update-stack.html

If you don't specify a value, CloudFormation uses the role that was previously associated with the stack. If no role is available, CloudFormation uses a temporary session that is generated from your user credentials.

Before --role-arn was being specified, CFN was indeed using a session based on the user credentials and had all permissions. However, after the first update where --role-arn NotAdmin was specified, CFN now remembers that role as the "stack role." Subsequent updates that don't specify --role-arn default to using that role, hence the permission problems.

As to why RoleWithAdminAccess cannot be assumed -- that's because using the role via --role-arn as opposed to the default session inheritance requires an explicit grant to Principal: {Service: "cloudformation.amazonaws.com"} that allows assuming the role