How to use multiple functions in twisted deferToThread?

Question

According to documentation on twisted deferToThread (http://twistedmatrix.com/documents/current/api/twisted.internet.threads.deferToThread.html) I can give it one function and it's arguments.

I want to limit the output of the documents I will find (say to 3 documents), so I also want to use the limit function from MongoDB driver (pymongo).

Answer 1

"find" creates a PyMongo Cursor, and does no more work. "find" does not send a message to the MongoDB server, and it does not retrieve any results. The work does not begin unless you iterate the cursor like this:

for doc in cursor:
    print(doc)

Or:

all_docs = list(cursor)

So the way you're doing it is already wrong: you're deferring to a thread the work of creating a Cursor, which does not need to be deferred because it doesn't do network I/O. But you're then using the cursor on the main thread, which you do need to defer.

So I propose something like:

def find_all():
    # find_one() actually does network I/O
    doc1 = self.mongo_pool.database[collection].find_one(self.my_id)
    # creating a cursor does no I/O
    cursor = self.mongo_pool.database[collection].find().limit(3)
    # calling list() on a cursor does network I/O
    return doc1, list(cursor)

stuff_deferred = deferToThread(find_all)