GAE/P: Implementing Exponential backoff for RPC calls

Question

I know that exponential backoff is a good thing when RPC calls fail. So far in my GAE/P app I have implemented exponential backoff by using the task queue:

deferred.defer(function_that_makes_RPC_call)

If the function that does the RPC call raises an exception, then the exponential backoff of the task queue takes care of it, and I don't have to worry about it.

A problem, however, is that deferred.defer is itself an RPC call that can fail! I sometimes get this error:

DeadlineExceededError: The API call taskqueue.BulkAdd() took too long to respond and was cancelled.

So it seems I can no longer be lazy and have to implement my own exponential backoff. :(

I'm thinking to put a wrapper around deferred.defer that implements exponential backoff using backoff, like this:

@backoff.on_exception(backoff.expo,
                      (exception1, exception2, ...),
                      max_tries=8)
def defer_wrapper(function_that_makes_RPC_call):
    deferred.defer(function_that_makes_RPC_call)

Here, the decorator implements the backoff where a retry happens when one of the enumerated exceptions (e.g., exception1, exception2, ...) is raised.

A couple questions about this:

1. Is this a good solution for implementing exponential backoff?
2. What exceptions do I need to list? Anything other than DeadlineExceededError?

I know it is somewhat redundant to have my own exponential backoff and then submit to a task queue, but I'm thinking that deferred.defer should fail more rarely than other RPC calls and I'd like to respond to the request ASAP.

Answer 1

In particular for the DeadlineExceededError in attempts to enqueue a deferred task I'd just do back2back retries instead of using an exponential backoff - the attempts will be spaced 5s apart due to the deadline interval expiration itself anyways, which gives a max of 12 retries before the request itself hits its deadline.

Probably a good idea for other types of failures, though.