Why do I get ignored exceptions when using a tailable MotorCursor and closing the Motor client connection?

Question

I wrote the following simplified version of my code:

from sys import exit
from tornado.ioloop import IOLoop
from tornado.gen import coroutine
from pymongo.errors import CollectionInvalid
from motor import MotorClient


client = MotorClient()
db = client.db_test
coll_name = 'coll_test'
coll = db[coll_name]
cursor = None


@coroutine
def stop():
    yield cursor.close()
    client.disconnect()
    IOLoop.current().stop()
    exit()


@coroutine
def create_cursor():
    global cursor

    try:
        yield db.create_collection(coll_name, capped=True, size=1000000)

    except CollectionInvalid:
        print('Database alredy exists!')

    yield coll.save({})
    yield coll.save({})
    cursor = coll.find(tailable=True, await_data=True)
    yield cursor.fetch_next
    cursor.next_object()

if __name__ == "__main__":
    IOLoop.current().spawn_callback(create_cursor)
    IOLoop.current().call_later(10, stop)
    IOLoop.current().start()

When I run it, I randomly get none or one of this two errors:

Exception ignored in: <bound method MotorCursor.__del__ of MotorCursor(<pymongo.cursor.Cursor object at 0x7fd3a31e5400>)>
Traceback (most recent call last):
  File "./env/lib/python3.4/site-packages/motor/__init__.py", line 1798, in __del__
TypeError: 'NoneType' object is not callable

Exception ignored in: <bound method MotorCursor.__del__ of MotorCursor(<pymongo.cursor.Cursor object at 0x7f4bea529c50>)>
Traceback (most recent call last):
  File "./env/lib/python3.4/site-packages/motor/__init__.py", line 1803, in __del__
  File "./env/lib/python3.4/site-packages/motor/__init__.py", line 631, in wrapper
  File "./env/lib/python3.4/site-packages/tornado/gen.py", line 204, in wrapper
TypeError: isinstance() arg 2 must be a type or tuple of types

I'm using Python 3.4.3, Tornado 4.1, Pymongo 2.8, Motor 0.4.1 and MongoDB 2.6.3.

This problem only appears when the tailable and await_data options are True at cursor's creation.

When I don't close the cursor I also get Pymongo's errors. But I think I should explicitly close it because it's a tailable cursor.

I've googled it but I had no luck. Any suggestions?

Answer 1

This was an unknown bug in Motor, I've tracked and fixed it at MOTOR-67. You observed a couple problems.

First, the Motor cursor's destructor had a bug where it would try to send a "killcursors" message to the MongoDB server, even after you'd called close. You closed the cursor, disconnected the client, and exited the Python interpreter. During interpreter shutdown, the cursor is destroyed and tries to send "killcursors" to the server, but the client is disconnected so the operation fails and logs a warning. This is the bug I've fixed and will release in Motor 0.6.

You call exit() from within a function that has a reference to a cursor, so the cursor's destructor runs during interpreter shutdown. The shutdown sequence is complex and unpredictable; often, the destructor runs after the greenlet module is destroyed. When the cursor destructor calls greenlet.getcurrent() at line 1798, the getcurrent function has already been set to None, hence "TypeError: 'NoneType' object is not callable".

I recommend not calling "exit()" from within a function. Your call to IOLoop.current().stop() allows the start function to return, and the interpreter to exit gracefully.