How to run canaries that depend on data to be already populated on AWS?

This is a high level architectural question, so no code samples should be needed.

I have a TS CDK application that deploys my infrastructure to AWS. The infra contains a number of lambdas. The lambdas rely on CodeDeploy to deploy them with canaries. The canaries trigger the new lambda pre-traffic.

Now for the tricky part.

The lambdas rely on AWS Secrets Manager. The infra defines the Secret Manager instances, but not the actual values. This means that the lambda is doomed to fail until somebody goes in there and manually enters the secret value. Meanwhile, the canaries will be triggered way before I can enter these values, resulting in the rollback of the entire deployment. I feel like adding some flags to my CDK context that would allow me to skip the canaries could alleviate the problem, but then I don't actually know if the deployment was successful and I might end up deploying random crap that doesn't work at all.

How do I find a way out of this race condition?

Solution

You can separate your deployment stages so that the secrets are deployed first, get populated, and then the infra that depends on them is deployed. An example:

A stack that contains the secrets is deployed. It exports the secrets' full ARN(s) by using CfnOutput: link
You add a stage where you automate setting the values, for example with AWS CLI. You use the secret ARN(s) from the 1st stage to do so
You deploy the lambdas. You can import the secrets into this stack by using the Secret.fromSecretCompleteArn() static method: link. This is usually enough to do basic things like Secret.grantRead().

There might be different impediments preventing you from doing it like this, but that would be difficult to understand without intimate knowledge of your setup.