I am trying to diagnose an issue where I am hitting service limits on ECS. My question is, how do I diagnose a service limit issue to figure out exactly which service limit I need to increase?
The error that I ran into doesn't really provide any details on which limit I hit, just that it was ECS:
Rate exceeded (Service: AmazonECS; Status Code: 400; Error Code: ThrottlingException; Request ID: 48f19207-b486-44fb-8cec-3d5e694392e6; Proxy: null)
Another thing to note, is that this is ECS on Fargate so wondering if I am possibly exceeding a Fargate quota.
I've reviewed the ECS service quotas for my account and it doesn't seem like I should be hitting one.
Is there a way I can find out exactly which limit I hit?
That's not a service quota error, that's an API call rate limit issue. It says "Rate exceeded" not "Quota exceeded". The rate is the amount of times you are calling ECS to create tasks within a given time period. Please see the documentation on the ECS API rate limits.
I am getting this error when launching hundreds of ECS tasks at once
Per the documentation I linked, the RunTask API (when launching EC2 backed tasks) has a maximum burst rate of 100 requests per second, and a sustained rate of 40 per second. When launching to AWS Fargate, there is a hard limit of 20 requests per second.
You can request an increase for API throttling quotas for your AWS account via the AWS Support Center.
Amazon recommends implementing an exponential backoff algorithm for API calls that may encounter throttling. Amazon has posted some articles and examples of implementing this algorithm. Those are linked at the bottom of the documentation I linked.