This is a probably basic question, but I have not been able to find the answer.
I have a long-running process that produces data every few minutes that I would like the client to receive as soon as it is ready. Currently I have the long-running process in a Task Queue, and it adds channel messages to another Task Queue from within a for loop. The client successfully receives the channel messages and downloads the data using a get request; however, the messages are being sent from the task queue after the long-running process finishes (after about 10 minutes) instead of when the messages are added to the task queue.
How can I have the messages in the task queue sent immediately? Do I need to have the for loop broken into several tasks? The for loop creates a number of dictionaries I think I would need to post to the data store and then retrieve for the next iteration (does not seem like an ideal solution), unless there is an easier way to return data from a task.
When I do not add the messages to a Task Queue and send the messages directly in the for loop, the server does not seem to respond to the client's get request for the data (possibly due to the for loop of the long-running process blocking the response?)
Here is a simplified version of my server code:
from google.appengine.ext import db
from google.appengine.api import channel
from google.appengine.api import taskqueue
from google.appengine.api import rdbms
class MainPage(webapp2.RequestHandler):
def get(self):
## This opens the GWT app
class Service_handler(webapp2.RequestHandler):
def get(self, parameters):
## This is called by the GWT app and generates the data to be
## sent to the client.
#This adds the long-process to a task queue
taskqueue.Task(url='/longprocess/', params = {'json_request': json_request}).add(queue_name='longprocess-queue')
class longprocess_handler(webapp2.RequestHandler):
def post(self):
#This has a for loop that recursively uses data in dictionaries to
#produce kml files every few minutes
for j in range(0, Time):
# Process data
# Send message to client using a task queue to send the message.
taskqueue.Task(url='/send/', params).add(queue_name=send_queue_name)
class send_handler(webapp2.RequestHandler):
def post(self):
# This sends the message to the client
# This is currently not happening until the long-process finishes,
# but I would like it to occur immediately.
class kml_handler(webapp2.RequestHandler):
def get(self, client_id):
## When the client receives the message, it picks up the data here.
app = webapp2.WSGIApplication([
webapp2.Route(r'/', handler=MainPage),
webapp2.Route(r'/Service/', handler=Service_handler),
webapp2.Route(r'/_ah/channel/<connected>/', handler = connection_handler),
webapp2.Route(r'/longprocess/', handler = longprocess_handler),
webapp2.Route(r'/kml/<client_id>', handler = kml_handler),
webapp2.Route(r'/send/', handler = send_handler)
],
debug=True)
Do I need to break up the long-process into tasks that send and retrieve results from the data store in order to have the send_handler execute immediately, or am I missing something? Thanks
The App Engine development server only processes one request at a time. In production, these things will occur simultaneously. Try in production, and check that things behave as expected there.
There's also not much reason to use a separate task to send the channel messages in production - just send them directly from the main task.