Why is aws lambda getting "AccessDeniedExceptionKMS" Error Message?

Question

I just deployed a lambda (using Terraform from gitlab runner) to a new aws account. This pipeline deploys a lambda to another (dev/test) account without issues, but when I try to deploy to my prod account, I get the following error:

I'm honing in on the statement, "The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access."

I have confirmed that the encryption config for the env vars are set to use default aws/lambda key instead of a customer master key. That seems to contradict the language of the error which refers to a customer master key...?

The role assumed by the lambda does have a policy which includes two kms actions:

        "Sid": "AWSKeyManagementService",

        "Action": [

            "kms:Decrypt",

            "kms:DescribeKey"

        ]

By process of elimination, I wonder if the issue is a lack on the part of the resource-based policy on the kms key. Looking in the kms keys, under aws managed, I find the aws/lambda key has the following key policy:

{
    "Version": "2012-10-17",
    "Id": "auto-awslambda",
    "Statement": [
        {
            "Sid": "Allow access through AWS Lambda for all principals in the account that are authorized to use AWS Lambda",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:CreateGrant",
                "kms:DescribeKey"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "kms:ViaService": "lambda.us-east-1.amazonaws.com",
                    "kms:CallerAccount": "REDACTED"#<-- Account where lambda deployed
                }
            }
        },
        {
            "Sid": "Allow direct access to key metadata to the account",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::REDACTED:root"#<-- Account where lambda deployed
            },
            "Action": [
                "kms:Describe*",
                "kms:Get*",
                "kms:List*",
                "kms:RevokeGrant"
            ],
            "Resource": "*"
        }
    ]
}

This is very puzzling. Any pointers appreciated!

Answer 1

This was solved by simply deleting the lambda and then re-running my pipeline to re-deploy it. All I can conclude is that something was corrupted in the first deployment.