Search code examples
mongodbfastapimotordriver

Mongo Motor Client AllowDiskUse

I am currently using the Motor client (v3.1.1) in a FastAPI application. When attempting to implement pagination, the following code works fine:

client = motor.motor_asyncio.AsyncIOMotorClient('mongodb://<my-connection-string>/')
database = client.test_database
collection = database.test_collection
pipeline = [
    { "$facet": 
        {"items": 
            [{ "$match": { }},
            {"$sort": {"_id": -1}},
            { "$skip": pageskip },
            { "$limit": pagesize },
            ],
        "totalCount": [{ "$count": "count" }]}}
        ]

cursor = collection.aggregate(pipeline)

However after a certain number of pages I get the following error:

raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: Sort exceeded memory limit of 104857600 bytes, but 
did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt 
in., full error: {'ok': 0.0, 'errmsg': 'Sort exceeded memory limit of 104857600 
bytes, but did not opt in to external sorting. Aborting operation. Pass 
allowDiskUse:true to opt in.', 'code': 292, 
'codeName':'QueryExceededMemoryLimitNoDiskUseAllowed'}

I tried using [allowDiskUse][1] directly in aggregation but then I get the following:

raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: Unrecognized pipeline stage name: 'allowDiskUse', 
full error: {'ok': 0.0, 'errmsg': "Unrecognized pipeline stage name: 'allowDiskUse'", 
'code': 40324, 'codeName': 'Location40324'}

Thanks in advance if anyone knows how to implement allowDiskUse with the Motor client ...

Solution

  • You add it as flag to your aggregate method:

    cursor = collection.aggregate(pipeline, allowDiskUse=True)