How do I poll a web service from a GAE service in short intervals?

Question

I'm developing a client app that relies on a GAE service. This service needs to get updates by polling a remote web service on a less than 1 minute interval so cron jobs are probably not the way to go here.

From the GAE service I need to poll the web service in intervals of a couple of seconds and then update the client app. So to break it down:

1. GAE service polls the remote web service in 5 sec intervals.
2. If a change is made, update the client app instantly.

Step 2 is solved already, but I'm struggling to find a good way on a polling of this sort. I have no control over the remote web service so I can't make any changes on that end.

I've looked at the Task queue API but the documentation specifically says that it is unsuitable for interactive applications where a user is waiting for the result

How would be the best way to solve this issue?

Answer 1

Use cron to schedule a bunch of taskqueue tasks with staggered etas

def cron_job():  # scheduled to run every 5 minutes
    for i in xrange(0, 60*5, 5):
        deferred.defer(poll_web_service, _countdown=i)

def poll_web_service():
    # do stuff

Alternatively, with this level of frequency, you might as well have a dedicated instance on this. You can do this with manual-scaling microservice and you can have the request handler for /_ah/start/ never return, which will let it run forever (besides having periodic restarts). See this: https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed#instance_scaling

def on_change_detected(params):
    queue = taskqueue.Queue('default')
    task = taskqueue.Task(
        url='/some-url-on-your-default-service/',
        countdown=0,
        target='default',
        params={'params': params})
    queue.add(task)

class Start(webapp2.RequestHandler):

    def get(self):
        while True:
            time.sleep(5)
            if change_detected:  # YOUR LOGIC TO DETECT A CHANGE GOES HERE
                on_change_detected()

_routes = [
    RedirectRoute('/_ah/start', Start, name='start'),
]

for r in _routes:
    app.router.add(r)