Glue Job Cross-Account secret access failing despite policies

Question

Note: I have reviewed other questions and believe this is unique because it specifically pertains to cross-account secrets access using a glue job.

I am having an issue where a glue job, assuming a service role in one account, is unable to access a secret stored in another account, despite my policies which I believe should allow it to do so.

Error I'm seeing (with redacted values): An error occurred (AccessDeniedException) when calling the GetSecretValue operation: Access to KMS is not allowed. This version of secret is not encrypted with the current KMS key.

Question

What about the below setup is causing the permission failure?

Diagram of Current Situation

My Understanding of What Needs to be Done

Based on the AWS docs and a 2018 blog post, I think what we have to do is:

• Add a Policy on the secret to allow access to service role
• Add a Policy on the CMK to allow service role to decrypt
• Add a Policy on Service role to allow access to secret
• Add a Policy on Service role to allow CMK decryption

Current Policies

Policy on Secret

{
  "Version" : "2012-10-17",
  "Statement" : [ {
    "Effect" : "Allow",
    "Principal" : {
      "AWS" : "arn:aws:iam::GLUE_ACCOUNT:role/GLUE_SERVICE_ROLE"
    },
    "Action" : "secretsmanager:GetSecretValue",
    "Resource" : "*",
    "Condition" : {  
      "ForAnyValue:StringEquals" : {
        "secretsmanager:VersionStage" : "AWSCURRENT"
      }
    }
  } ]
}

Policy on CMK

Note that the redacted SECRET_NAME below contains the few characters that AWS appends in the ARN, which it seems we need to include.

{
    "Sid": "AllowUseOfTheKey",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::GLUE_ACCOUNT:role/GLUE_SERVICE_ROLE"
    },
    "Action": ["kms:Decrypt","kms:DescribeKey"], 
    "Resource": "*",
    "Condition": {
        "StringEquals": {
            "kms:ViaService": "secretsmanager.us-east-1.amazonaws.com"
        },
        "StringLike": {
            "kms:EncryptionContext:SecretARN": "arn:aws:secretsmanager:us-east-1:SECRET_ACCOUNT:secret:SECRET_NAME"
        }
    }
}

Policy Statements Attached to Glue Service Role in Glue Account

{
    "Sid": "AllowGetSecretValue",
    "Effect": "Allow",
    "Action": [
        "secretsmanager:GetSecretValue"
    ],
    "Resource": [
        "arn:aws:secretsmanager:us-east-1:SECRET_ACCOUNT:secret:SECRET_NAME"
    ]
},
{
    "Sid": "AllowKMSDecrypt",
    "Effect": "Allow",
    "Action": [
        "kms:Decrypt"
    ],
    "Resource": [
        "arn:aws:kms:us-east-1:SECRET_ACCOUNT:key/CMK_KEY_ID"
    ]
},

Trust policy on the service role

Just to confirm that the glue job does indeed seem to have the authority to assume the service role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "glue.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Possible Next Steps

As thought experiments, I'm considering doing the below just to see if it works:

• Removing the condition in the secret policy
• Removing the conditions in the CMK policy
• Adding kms:DescribeKey to the Service Account policy as well, though I don't think that's going to resolve things.

Answer 1

As sometimes happens with these involved questions, the piece of information to answer the question was found outside of the information I'd provided, despite my best efforts.

The solution was two-fold:

• The authorization error that was part of the original question wasn't due to authentication access at all. It was because the code that used boto3 specified the name of the secret instead of the full ARN. Of course secrets manager wouldn't magically know that secret was in a different account.
  • A surprising thing in this case is that AWS threw an unauthorized exception, and not a not found exception, which would have been infinitely more useful to our efforts.
• Once that was fixed, we saw the error that is currently in the ticket, around encrypting the CMK. Apparently when we changed the CMK, we did not select to automatically create a new version of the secret with that CMK. A simple edit and save of the secret by an authorized user was enough to re-encrypt it with the chosen CMK and resolve the issue.