AWS Secrets Manager and Cloud Formation - can not create secret because it already exists

Question

I have a CF template with a simple secret inside, like this:

Credentials:
    Type: 'AWS::SecretsManager::Secret'
    Properties:
    Name: !Sub ${ProjectKey}.${StageName}.${ComponentId}.credentials
    Description: client credentials
    SecretString: !Sub 
        '{"client_id":"${ClientId}","client_secret":"${ClientSecret}"}'

The stack is created successfully and the secret is correctly generated.

However when I delete the stack and recreate it again I get the following error message:

The operation failed because the secret pk.stage.compid.credentials already exists. (Service: AWSSecretsManager; Status Code: 400; Error Code: ResourceExistsException; Request ID: ###)

I guess this is because the secret is not really deleted but only marked for deletion for x days.

It is possible to delete a secret immediately via CLI, but how can this be done within the CF Template?

I need to delete and recreate the stacks because it is part of a continous integration/delivery pipeline which is automatically triggered on source code commits.

Answer 1

Normally when you delete a stack the secret should be deleted also; and CFN does the aforementioned immediate delete. This should succeed even if the secret was scheduled for deletion outside of the CFN stack.

If (after your stack was deleted) the secret was created by another cloud formation stack or the same test running in another CI pipeline re-created the secret, you might see this error. Also, most AWS systems (Secrets Manager included) are eventually consistent, and you may see a delay between the stack being deleted and the actual secret deletion. If your tests run quick enough, or the same secret name is re-used in multiple tests, the previous delete may not have completed before the next create.

We have faced similar problems in our CI stacks and the way we work around it is to use a per test random name that is generated. You could, for example, pass in a random prefix to your stacks as a parameter and use that to construct the name (ensuring each test uses a unique suffix).

BTW - you can test if a secret was scheduled for deletion or is actually not there by running get-secret-value on the secret. If it is scheduled for deletion you will see the error "...You can’t perform this operation on the secret because it was deleted", whereas if the secret is actually deleted you will see "Secrets Manager can’t find the specified secret". If you schedule a secret for deletion and then delete it with --force-delete-without-recovery you may see a short multi-second lag between the two states.