Search code examples
pythontwistedpymongo

Do you need to defer .count() in pymongo and twisted?


I am running twisted with pymongo. I know that when using pymongo .find() query you need to defer it when you iterate over the cursor.

Does the same apply to .count()? Do I need to defer it or is it non-stopping?

EDIT: if it has to be deferred, what would be the proper way of doing it?

Do I need to create a cursor first and then call count on it:

value_deferred = deferToThread(
                mongo_collection.find,
                mongo_query,
            )
value_deferred.count()

or is there any way to get the count right away?

If I do this:

def get_filtered_count():
    return db_collection.find(mongo_query).count()

value_to_get = deferToThread(get_filtered_count())

I get this error: exceptions.TypeError: 'int' object is not callable

EDIT 2: Is the use of yield here justified? I get errors when calling it otherwise.

@inlineCallbacks
def render_deferred(self, request):
  cursor = self.mongo.find()
  get_counter = yield deferToThread(cursor.count)
  page_size = 3
  number_of_pages = get_counter / page_size
  return final_value

def render_GET(self, request):
  ## some code
  deferred = self.render_deferred(request)

  deferred.addCallback(_send, request)
  deferred.addErrback(handle_failure, request)

Solution

  • What you must defer is the actual call to count. The find method just creates a cursor and does no I/O, whereas calling count does. So you can call find on the main thread or the worker thread, depending on what's convenient.

    When you call deferToThread, pass the function, don't call it:

    value_to_get = deferToThread(get_filtered_count)
    

    Or just:

    cursor = collection.find(query)
    deferred = deferToThread(cursor.count)