I'm working on a project consists of an api application (AWS API Gateway + AWS Lambda) and a database (AWS DynamoDB). I use Terraform and GitLab CI to automate deployment for this project. When I modify the API Gateway config or Lambda function to fix bugs, I need to re-deploy the whole system (probably including DynamoDB) by executing terraform apply. This may cause data loss because DynamoDB may be re-created. I want to release a new version api without changing database in CI/CD pipeline. In my understanding, terraform apply -target=xxx is suitable for a small system, but not suitable for a complex one.
I wonder if there any best practices for database provisioning using IaC. Is it better to treat database separately and tracked in an entirely different system? Do I need to implement database backup and migration in my CI/CD pipeline?
Generally when using IaC, Database is not something which gets modified as frequently as application deployments. Even when you deploy (update) Lambda + API-Gateway, you're not touching DB and terraform apply should not modify the "connection" between App layer and Data layer. Even if it does, try not to "re-create" database and pass all the configs from IaC itself.
If your use-case has to destroy database and re-create then you have to automate backup of data somewhere and load it into DB after provisioning. This is going to be a pain in the back very soon. Recreating database is not ideal, backups take time, and reloading data is costly (data transfers) and SLOW!
Maybe you should re-architect your IAC strategy where you don't have to re-create DB a lot. Updating small configs should not be a problem as configs will be passed around via IaC (app layer to DB layer & vice versa)