AWS Parameter Store : AWSSimpleSystemsManagementException: Rate exceeded

Question

At the moment, I need to retrieve the last 10 values of some parameters in AWS Parameters Store

I am using the following code in kotlin :

    val p1 = retrieveAllValidVersions("P1")
    val p2 = retrieveAllValidVersions("P2")
    val p3 = retrieveAllValidVersions("P3")

Here is the code of the retrieveAllValidVersions

    private fun retrieveAllValidVersions(paramName: String): List<ParameterHistory> {
        val res = mutableListOf<ParameterHistory>()
        val ssmClient = AWSSimpleSystemsManagementClientBuilder.defaultClient()
        var nextToken : String? = null
        do {
            val ssmParams = ssmClient.getParameterHistory(GetParameterHistoryRequest()
                    .withName(paramName)
                    .withWithDecryption(true)
                    .withNextToken(nextToken)
            )
            res.addAll(ssmParams.parameters)
            nextToken = ssmParams.nextToken
        } while (nextToken != null)


        return validVersions.sortedByDescending { it.version }.take(10)
    }

As described in https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_ssm, the maximum number of versions for a parameter is 100

And as described by https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_GetParameterHistory.html, you can only retrieve 50 values in maxResults so I need to make 2 calls for each parameter (because I have more than 50 versions)

So each retrieval of my 3 parameters costs 6 queries to SSM

I cache the last 10 values of each param in memory for 5 minutes

The problem is that when multiple instances of my lambdas expire their cache at the same time, they do the retrieval at the same time and

com.amazonaws.services.simplesystemsmanagement.model.AWSSimpleSystemsManagementException: Rate exceeded (Service: AWSSimpleSystemsManagement; Status Code: 400; Error Code: ThrottlingException; Request ID: xxx)

For ex, if 3 instances are concerned, that will do 18 requests in less than a second and I will hit the error (note : I do not know exactly if the number of instances that hit this code at the same time is 3, this is just a guess to illustrate that at some point you hit the error)

So I have 2 questions :

First, is there a way to retrieve last versions of a parameter first ?

That way, I will then do half of the requests so I will hit the problem less often !

Second, how do I automatically retry a throttling error ?

I have find this AWS blog [1] post saying that I have to parse the error message but this is an old post (2013) and this is very ugly (the moment AWS changes the message, all the mechanism collapses) !

[1] : https://aws.amazon.com/fr/blogs/messaging-and-targeting/how-to-handle-a-throttling-maximum-sending-rate-exceeded-error/

A final note : I am using the parameters store with "auto-encryption" and IAM, I do not want to store the parameters in my own database nor to cache them in a shared memory cache like redis !

Answer 1

I've found a solution / workaround that I will share here.

Please feel free to comment if you find a better way !

TL;DR : the workaround works because it does only 1 request to SSM (per lambda) instead of 6 thanks to GetParametersByPath recursive instead of GetParameterHistory.

So, to simplify, my use case was to store a secret to crypt a token and to be able to use it for 10 hours.

Note : IRL, I use 3 different secrets, hence the P1, P2, P3 in the question. In the following, I will simplify and only talk about 1 secret as it works similarly for any number of secrets (until you reach the maximum number of parameters in SSM which is 10K ...)

It previously works like this : I rotate the secret every hour, I decrypt the token against the last 10 versions -> if a user sends a token that is 11 hours old or more, I cannot decrypt it anymore.

Now, instead of having a single parameter with multiple versions, I have multiple parameters with only the last version of each being safe to use.

My SSM Parameter Store previously looked like (for each parameter)

/secret/P1

And now it looks like

/secrets/P1/s1 -> a secret
/secrets/P1/s2 -> a secret
...
/secrets/P1/s10 -> a secret
/secrets/P1/current -> s4 

My code now does GetParametersByPath("/secrets") with recursive to retrieve in one request all the valid secrets for all parameters (i.e P1, P2, P3).

So each lambda does one request, there is little chance to hit RateExceeded.

When a client sends a token, I try to decrypt against the 10 current secrets.

The secret rotation has changed to : retrieve /current and change the next one (i.e if current is s4, we change s5 and set /current to s5)

As a final note, I also implemented an improvement which is to add /current into the token (unencrypted).

Doing that, I do not need to check against the 10 secrets, only against the one contained in the token.

Note that I could have done that improvement before (by sending the version number unencrypted in the token), I just did not think of it before.

Hope that helps someone