My app engine app has 2 queues. A worker from the first queue initiates several workers in the second queue and wait for them to complete and consolidate.
This works fine when I deploy it, but it doesn't work never spawns the tasks in the second queue when testing locally. It gets stuck while waiting for it to update (see code below). The reason seems to be that dev_appserver is single threaded and does not launch services in the background.
This is really slowing down my development since I have to deploy to the cloud to test anything.
Any ways around this?
Edit: Adding the (sort of) pseudo code below. Again, this works perfectly fine in the cloud because I have a 10 minute time limit to complete the task and each BigTask takes not more than 30-40 seconds to complete.
class BigTask(webapp2.RequestHandler):
def do_stuff(self):
#do something here
small_task1 = taskqueue.add(...)
small_task2 = taskqueue.add(...)
small_task3 = taskqueue.add(...)
small_task4 = taskqueue.add(...)
# Create ndb entries for all small tasks
while True:
time.sleep(2)
# Check if ndb entry updated for all small_tasks
if status == 'Completed': #for all
break
add_results(...) # of all tasks here
# Update ndb entry for big_task
# done
class SmallTask(webapp2.RequestHandler):
def do_stuff(self):
# Do processing
# Update ndb entry
# done
Generally sleeping in GAE app code is not a good idea - wasting instance uptime, increasing request latency, risk of exceeding the request deadline etc.
You can achieve the same functionality by enqueuing a delayed BigTask (using the countdown option) which would check the status for all smaller tasks, do the add_results(...) stuff if they completed or, if not, re-queue itself (again delayed) to re-check later on.
As a side effect you could even use the same task queue for both kind of tasks (if that's inline with your app requirements, of course).