Search code examples
pythontornado

Tornado's IOLoop.call_later is behaving inexplicably

From an interpreter session:

>>>io_loop.call_later(comments_page_delay, functools.partial(http_client.fetch, "google.com", lambda x: print('kek')))
<tornado.ioloop._Timeout object at 0x7fe9a2427b08>
>>> io_loop.start()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/tornado/ioloop.py", line 755, in start
    raise RuntimeError("IOLoop is already running")
RuntimeError: IOLoop is already running
>>> io_loop.call_later(comments_page_delay, functools.partial(http_client.fetch, "google.com", lambda x: print('kek')))
<tornado.ioloop._Timeout object at 0x7fe9a0267808>
>>> io_loop.stop()
>>> io_loop.start()
>>> io_loop.start()
kek
kek
kek

Here:

io_loop = tornado.ioloop.IOLoop.current()
comments_page_delay = 0.1

EDIT:

AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")
http_client = AsyncHTTPClient()
http_client.max_clients = max_clients
http_client.request_timeout = request_timeout

Problem:

The ioloop doesn't seem to run the call added with call_later. It's running, as evident by the error when I try to do io_loop.start(). To make it actually execute, I have to first stop the loop, then start it twice... Very confusing.

I admit it's been a while since I've used Tornado, but this is basically equivalent to other code I had which I know worked.

EDIT: The code works correctly on the first run, but not on consecutive runs. At that point the need to run io_loop.start() twice returns

Here's the full code (slightly censored):

"""...docstring..."""
import functools

import tornado
from tornado.httpclient import AsyncHTTPClient
from lxml import html as lh

from config import comments_page_delay, max_clients, request_timeout
from debug import log

AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")
http_client = AsyncHTTPClient()
http_client.max_clients = max_clients
http_client.request_timeout = request_timeout

io_loop = tornado.ioloop.IOLoop.current()

def x(y, proxy):
    """...docstring..."""
    def handle_response(response):
        if response.error:
            log('...message...'
                .format(uid, response.error), 1)
        else:
            print('Worked')
            data = response.body

    comments_url = "...valid url..."

    io_loop.call_later(comments_page_delay, functools.partial(http_client.fetch, comments_url, handle_response,
                                                              proxy_host=proxy['host'], proxy_port=proxy['port']))

x(1, {'host': None, 'port': None})

io_loop.start()
print('ok')
io_loop.stop()

I'm running the code in an Emacs python interpreter session, if that is somehow relevant. I'll try in a normal python session in a second.

EDIT2: Here's what happens if I run that code in a normal python interpreter:

$ python -i fetch.py
Worked


Traceback (most recent call last):
  File "fetch.py", line 34, in <module>
    io_loop.start()
  File "/usr/lib/python3.6/site-packages/tornado/ioloop.py", line 863, in start
    event_pairs = self._impl.poll(poll_timeout)
KeyboardInterrupt
>>> >>> >>> get_user_profile_link(1, {'host': None, 'port': None})
>>> 
>>> 
>>> io_loop.start()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/tornado/ioloop.py", line 755, in start
    raise RuntimeError("IOLoop is already running")
RuntimeError: IOLoop is already running
>>> io_loop.stop()
>>> io_loop.start()
>>> io_loop.start()
Worked

It appears that io_loop.start() is blocking? I didn't think it was, that might be the issue. I'll have to rethink the architecture of my whole program now.

Solution

  • An uncaught exception (in this case the KeyboardInterrupt) leaves the IOLoop in an undefined state. It is not possible to restart the IOLoop after such an exception has occurred (because it could come at any point and leave the IOLoop's internals inconsistent). In general it's assumed that the process will simply exit after a KeyboardInterrupt. It's possible to create a new IOLoop and start over with it, but you'll have to be careful to recreate all objects that depend on it.

    Unfortunately, this means that it's practically difficult to use Tornado with the interactive interpreter. My usual workflow for developing Tornado applications is to run a script with python -m tornado.autoreload and let the script re-run from scratch whenever I edit it.