Where to store configuration for an application store in AWS

Question

I am building an AWS application that uses EC2, lambda, API gateway, S3, SQS and other services.

Is there a mechanism to store global application settings that can be shared among many components?

I'd like to avoid having environment variables set for each component; instead, each component should read config from a config repository.

I tried with S3 config files, but buckets have unique URLs that cannot be guessed by other components. Storing config in RDS doesn't work since RDS instances also have unique names.

Can KMS be used for this? If yes, how do I store and retrieve the S3 bucket name into KMS? I only found a way to store a 256 bit encryption key.

Answer 1

AWS System Manager Parameter Store can be used to store the configurations.

The configurations can be consumed directly in you application using AWS APIs. AWS also gives packages to get the values from parameter store. Example: for dotnet core -- https://aws.amazon.com/blogs/developer/net-core-configuration-provider-for-aws-systems-manager/

In Parameter Store values can be stored as String, StringList or SecureString.

SecureString can be used to store passwords and this can be secured using inbuilt or user defined KMS keys. So KMS keys just provide an option (key) to secure our data stored in various AWS resources.

For storing more secure information AWS Secret Manager can be used.

Pricing:

• Parameter Store is free for standard parameters, while advanced parameters are charged $0.05 per advanced parameter per month.
• Secret manager is charged $0.40 per secret per month.

Standard parameters are enough to store parameters for normal usage. But be sure not to fetch the parameters too frequent and have a retry logic to fetch parameters from AWS parameter store as there is a rate limiting in place (AWS packages will handle retry logic internally, so we do not need to worry on that. It even gives an option to refresh the config values after a timespan). It is better to get all required config values at application startup and store it in in-memory cache.

Parameter Store and Secret Manager uses Naming in the form of path. This is useful to store environment specific configs and also to restrict access to the parameters based on path using policies.

Example:

• Parameter Name --> /AppName/Environment/ParameterName --> /app/dev/version, /app/prod/version (longer paths are allowed)
• Restrict based on path(name) policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowSSMAccess",
            "Effect": "Allow",
            "Action": [
                "ssm:PutParameter",
                "ssm:DeleteParameter",
                "ssm:GetParameterHistory",
                "ssm:GetParametersByPath",
                "ssm:GetParameters",
                "ssm:GetParameter",
                "ssm:DescribeParameters",
                "ssm:DeleteParameters"
            ],
            "Resource": "arn:aws:ssm:*:*:parameter/*"
        },
    ]
}