boto ElasticMapReduce throttling and rate limiting

Question

I've run into rate limting from Amazon EMR a few times via boto API with the following:

boto.exception.EmrResponseError: EmrResponseError: 400 Bad Request
<ErrorResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
  <Error>
    <Type>Sender</Type>
    <Code>Throttling</Code>
    <Message>Rate exceeded</Message>
  </Error>
  <RequestId>69d74a63-7de3-11e0-aafc-2b540b1e5f42</RequestId>
</ErrorResponse>

The operation is a one-time operation request the state of a jobflow, so there shouldn't be any rate-limiting involved. Has anyone else ran into this issue? Also, there doesn't seem to be much documentation on EC2 and EMR throttling/rate limiting...

Answer 1

Almost all (if not all) AWS APIs are rate limited. Even reading data puts a load on their services (some more than others) so they protect themselves by limiting the rate of requests each account is allowed to make. According to the AWS docs the recommended approach to dealing with a throttling response is to implement exponential backoff in your retry logic.