How to fix deferred tasks being sent to not initialized instance?

Question

My Google AppEngine website is using deferred tasks to do a lot of extra work. Sometimes the amount of deferred tasks cause a new instance to be created. When this happens, all the tasks that are delegated to this new instance fail because they get sent there before the instance has had a chance to initialize. If enough instances are already started up before the tasks come in, no issues occur.

Here is an example of one of the failed tasks:

Permanent failure attempting to execute task (/.../google/appengine/ext/deferred/deferred.py:327)
Traceback (most recent call last):
  File "/.../google/appengine/ext/deferred/deferred.py", line 318, in post
    self.run_from_request()
  File "/.../google/appengine/ext/deferred/deferred.py", line 313, in run_from_request
    run(self.request.body)
  File "/.../google/appengine/ext/deferred/deferred.py", line 153, in run
    raise PermanentTaskFailure(e)
PermanentTaskFailure: No module named django

Is there a way to tell the deferred system to wait until an instance is fully initialized before sending tasks to it? I'm losing hair over this problem.

Answer 1

GAE warmup requests are designed to do exactly that:

Loading your app's code to a new instance can result in loading requests. Loading requests can result in increased request latency for your users, but you can avoid this latency using warmup requests. Warmup requests load your app's code into a new instance before any live requests reach that instance.

You just need to:

• enable the warmup requests
• register a handler for them
• add the handler code