Are deferred tasks parallelized in google app engine?

Question

I have a CRON job which imports everyday a list of 20 fresh files from a S3 bucket to a GS bucket.

Here is my code:

import webapp2
import yaml
from google.appengine.ext import deferred


class CronTask(webapp2.RequestHandler):

    def get(self):
        with open('/my/config/file') as file:
            config_dict = yaml.load(file_config_file)
        for file_to_load in config_dict:
            deferred.defer(my_import_function, file_to_load)


app = webapp2.WSGIApplication([
    ('/', CronTask)
], debug=True)

My question: Is this loading job parallelised automatically by the queue process using the deferred.defer function, or shall I paralelize it myself?

If we are in the second case, what techniques can I use to parallelize this loading process efficiently?

Answer 1

It will depend on your module's scaling settings, and the queue's processing settings but by default tasks will execute with some parallelism.