How do I implement a non-blocking wait in tornado's autoreload reload_hook?

Question

I have a web page that I would like to reload when I make changes, however, I may have threads running that I don't want to be interrupted by the autoreload. What I have now works, in that the threads are able to finish, but after making a change, but before the reload takes place, the web page does not respond if I click on anything.

@tornado.gen.coroutine
def wait_on_threads():
    """generate for non-blocking sleep"""
    for k, t in webserver.app.config['test_threads'].items():
        if t and t.is_alive():
            yield tornado.gen.sleep(5)
        else:
            webserver.app.config['test_threads'][k] = None


def reload_hook():
    """wait for daemon threads to finish"""
    while any(webserver.app.config['test_threads'].values()):
        wait_on_threads()
    rollback()

...

if __name__ == '__main__':        
        application.listen(address=ip, port=80)
        tornado.autoreload.add_reload_hook(reload_hook)
        tornado.autoreload.start()
        try:
            tornado.ioloop.IOLoop.instance().start()
        except KeyboardInterrupt:
            cleanup_handler()
            webserver.app.logger.info('Exit success.')
            tornado.ioloop.IOLoop.instance().stop()

Before, I just had sleeps in the reload hook and that worked, but made the web page unresponsive when I would make a change. The threads would successfully complete before reload, and after reload, the web server would become responsive again. I have tried to replace the sleeps with a non-blocking method, and again it works, but it behaves exactly the same, in that the web server blocks and won't respond until after the reload.

Am I doing this incorrectly, or is the autoreload feature going to cause the web page to lose responsiveness in any case?

This is tornado Version: 4.2.1 and python 2.7.10

Answer 1

Tornado's autoreload feature is intended for development use, when you don't care about interrupting what's in progress. (If you delay the reload, then either the server is unavailable during the reload or you can continue to add work to the old process, delaying the restart indefinitely). To restart a running service without disruption requires a more sophisticated process in which the old and new services overlap for a brief period.

For example, you could run two separate (sets of) processes and instruct a load balancer to shift traffic from one to the other. Or you could call HTTPServer.stop() in the old processes to free up the port, and then leave them running while you also start the new processes.