How to create X tasks as fast as possible on Google App Engine

Question

We push out alerts from GAE, and let's say we need to push out 50 000 alerts to CD2M (Cloud 2 Device Messaging). For this we:

1. Read all who wants alerts from the datastore
2. Loop through and create a "push task" for each notification

The problem is that the creation of the task takes some time so this doesn't scale when the user base grows. In my experience we are getting 20-30 seconds just creating the tasks when there is a lot of them. The reason for one task pr. push message is so that we can retry the task if something fails and it will only affect a single subscriber. Also C2DM only supports sending to one user at a time.

Will it be faster if we:

1. Read all who wants alerts from the datastore
2. Loop through and create a "pool task" for each 100 subscribers
3. Each "Pool task" will generate 100 "push tasks" when they execute

The task execution is very fast so in our scenario it seems like the creation of the tasks is the bottleneck and not the execution of the tasks. That's why I thought about this scenario to be able to increase the parallelism of the application. I would guess this would lead to faster execution but then again I may be all wrong :-)

Answer 1

We do something similar with APNS (Apple Push Notification Server): we create a task for a batch of notifications at a time (= pool task as you call it). When task executes, we iterate over a batch and send it to push server.

The difference with your setup is that we have a separate server for communicating with push, as APNS only supports socket communication.

The only downside is if there is an error, then whole task will be repeated and some users might get two notifications.