asyncio.lock in a Tornado application

Question

I'm trying to write a asynchronous method for use in a Tornado application. My method needs to manage a connection that can and should be shared among other calls to the function The connection is created by awaiting. To manage this, I was using asyncio.Lock. However, every call to my method would hang waiting for the lock.

After a few hours of experimenting, I found out a few things,

1. If nothing awaits in the lock block, everything works as expected
2. tornado.ioloop.IOLoop.configure('tornado.platform.asyncio.AsyncIOLoop') does not help
3. tornado.platform.asyncio.AsyncIOMainLoop().install() allows it to work, regardless if the event loop is started with tornado.ioloop.IOLoop.current().start() or asyncio.get_event_loop().run_forever()

Here is some sample code that wont work until unless you uncomment AsyncIOMainLoop().install():

import tornado.ioloop
import tornado.web
import tornado.gen
import tornado.httpclient
from tornado.platform.asyncio import AsyncIOMainLoop
import asyncio
import tornado.locks


class MainHandler(tornado.web.RequestHandler):

    _lock = asyncio.Lock()
    #_lock = tornado.locks.Lock()

    async def get(self):
        print("in get")
        r = await tornado.gen.multi([self.foo(str(i)) for i in range(2)])
        self.write('\n'.join(r))

    async def foo(self, i):
        print("Getting first lock on " + i)
        async with self._lock:
            print("Got first lock on " + i)
            # Do something sensitive that awaits
            await asyncio.sleep(0)
        print("Unlocked on " + i)

        # Do some work
        print("Work on " + i)
        await asyncio.sleep(0)

        print("Getting second lock on " + i)
        async with self._lock:
            print("Got second lock on " + i)
            # Do something sensitive that doesnt await
            pass
        print("Unlocked on " + i)
        return "done"


def make_app():
    return tornado.web.Application([
        (r"/", MainHandler),
    ])

if __name__ == "__main__":
    #AsyncIOMainLoop().install()  # This will make it work
    #tornado.ioloop.IOLoop.configure('tornado.platform.asyncio.AsyncIOLoop')  # Does not help
    app = make_app()
    app.listen(8888)
    print('starting app')
    tornado.ioloop.IOLoop.current().start()

I now know that tornado.locks.Lock() exists and works, but I'm curious why the asyncio.Lock does not work.

Answer 1

Both Tornado and asyncio have a global singleton event loop which everything else depends on (for advanced use cases you can avoid the singleton, but using it is idiomatic). To use both libraries together, the two singletons need to be aware of each other.

AsyncIOMainLoop().install() makes a Tornado event loop that points to the asyncio singleton, then sets it as the tornado singleton. This works.

IOLoop.configure('AsyncIOLoop') tells Tornado "whenever you need an IOLoop, create a new (non-singleton!) asyncio event loop and use that. The asyncio loop becomes the singleton when the IOLoop is started. This almost works, but when the MainHandler class is defined (and creates its class-scoped asyncio.Lock, the asyncio singleton is still pointing to the default (which will be replaced by the one created by AsyncIOLoop).

TL;DR: Use AsyncIOMainLoop, not AsyncIOLoop, unless you're attempting to use the more advanced non-singleton use patterns. This will get simpler in Tornado 5.0 as asyncio integration will be enabled by default.