What my code needs a gen.sleep to execute a function async?

Question

I have the following code written based on Python's Tornado:

def process_data(data):
    # To something

def handler(message):
    if message['type'] == 'message':
        data = message['data']
        IOLoop.current().spawn_callback(process_data, data)

async def main():
    redis_client = RedisClient(redis_conf)
    pubsub = redis_client.subscribe("CHANNEL", handler)

    async def fetch_messages():
        while True:
            pubsub.get_message()
            await gen.sleep(0.0001) 


    await fetch_messages()

if __name__ == "__main__":
    import logging
    logging.basicConfig()

    parse_command_line()

    tornado.web.Application(debug=options.debug)

    io_loop = ioloop.IOLoop.current()
    io_loop.run_sync(main)

Given the above code I can see that process_data is being called. However, if I remove await gen.sleep(0.0001), process_data is never called. Does anyone know why?

Answer 1

IOLoop.spawn_callback(callback, *args, **kwargs)

Calls the given callback on the next IOLoop iteration.

If you're keep calling some synchronous code (while True without await) you don't return control to event loop and event loop can't make it's iteration to execute callback.

Line await gen.sleep(0.0001) - is where controls returns to event loop so it can do something (like executing callbacks).

Tornado has special object to return control to event loop for such cases as yours - gen.moment:

    while True:
        pubsub.get_message()
        await gen.moment

I didn't work with Tornado, but I bet that even better would be to use some redis client designed to be used in async programs, see this answer.