Has anyone noticed some unpredictable failures on AWS Secret Manager when trying to retrieve secret values? I'm using my own encryption key, and I've found that frequently, I am getting a "Failed to fetch a list of secrets" error on the AWS console after encrypting a secret. This seems to happen if I change the encryption key after an initial encryption, but it has happened without that as well.
I also think I've seen a case where the encryption key changed from a custom key to default without any action from me.
I've also seen an issue where two stacks set up nearly identically have an inconsistency where one can read an encryption key when calling Secrets Manager but one cannot. It looks like an IAM issue, but I haven't found any difference between the two stacks and their IAM settings. I only mention this in case it gives some clue to the issue above.
I am seeing the same thing as well after I changed the encryption key. I don't understand why this is happening. I will open a ticket with AWS and report back.
OK after talking to AWS Support the issue seems to be a bug. If you disabled (or marked it for deletion) your old encryption key than you will experience this issue.
To fix this you will need to cancel the deletion of your old encryption key, AND change its status to "Enabled". After this you will be able to retrieve your secrets using your new encryption key.
Unfortunately, this is the current workaround until AWS has a permanent solution.
Hope this helps.