We have a setup where RDS is provisioned through terraform, cloudwatch alarm for RDS' free storage space metric is set to <10% total RDS storage through terraform. Total RDS storage is a static value. Whenever there's storage shortage we manually increase the RDS storage through AWS console. Is there way to update the cloudwatch alarm threshold automatically, to set it to 10% of new total storage after increasing RDS storage?
Although considered an anti-pattern, there is an alternative. It is considered an anti-pattern because it is generally a bad idea to have two different processes controlling the configuration of the same resource. However, if you insist on controlling the size of the RDS outside the terraform plan, you can automate the cloudwatch metric settings.
The root problem is that the FreeSpaceAvailable RDS metric only understands bytes, not percentages. Terraform does that conversion for you so you don't see it. That is why the alarm is not changing when you change the allocated space in the RDS console. You could develop a simple lambda function that uses one the of the AWS APIs (like boto3 for python) to periodically query the size the RDS database and update the alarm on FreeSpaceAvailable accordingly. It could be even more sophisticated by having SNS notify a SQS queue about changes in RDS configuration which could then trigger the lambda to evaluate the metric, which would make the update very fast and efficient. You would need to apply a IAM policy which allows the lambda to read data on the RDS instance in question and permission to update the metric in question. Alternatively, you could have it be manually run and then run it right after the AWS console update, but in that case you could probably just as easily manually update the metric yourself.
I still wouldn't recommend this be the path due to the control of an already controlled resource by terraform.