MongoDB performance issue, probably due to projection getting ignored in NodeJS project, using native MongoDB driver

Question

Long version: (Scroll down for TLDR)

I have a collection 'libraries'. A library may be of type "template" or "standard-template" (in this scenario). If it is owned by a organization it will contain a "organization_id" (which is otherwise null), if it is unowned but a organization has access to it this id will be added to a "organizations" array. Libraries with type "template" are always owned, those with type "standard-template" are never owned. The query to do so looks like:

{
    "$or": [{
        "organization_id": id,
        "type": "template"
    }, {
        "organization_id": null,
        "type": "standard-template",
        "organizations": id
    }]
}

I have a index like {"organization_id": 1, "type": 1} and there aren't many "standard-template" libraries. A explain will tell me that this query takes +- 4ms to execute and returns 50 documents.

In my NodeJS application it will take about 12 seconds. This is likely due to the size of each document (which may vary from a few KB to 10MB).

I'm trying to limit this using a projection to only receive the relevant fields, this however, appears to be ignored entirely. I've tried all kinds of variations of the code below, but none seemed to make a difference.

TLDR

The projection value in my code seems to be ignored. In the example I attempt to retrieve only "_id" fields, yet end up with the entire documents.

Code used for testing

let id = ObjectID('5e56503cafc87b893b92827c');
let start = performance.now();
let find = mongodb.collection('libraries').find(
    {
        "$or": [
            {"organization_id": id, "type": "template"},
            {"organization_id": null,"type": "standard-template", "organizations": id}
        ]
    }, {_id: 1});

while (await find.hasNext()) {
    const doc = await find.next();
    console.log(doc); //HUGE! way more than just _id!
}

console.log(performance.now() - start); //about 12000 ms

Even shorter test code:

console.log(await mongodb.collection('libraries').findOne({}, {_id: 1})); //HUGE!

In any examples or docs I found it is done seemingly the same way. I'm pretty sure I've done it this way in the past. Am I overseeing something? Any insights are very much appreciated.

Answer 1

Since you are using cursor to iterate through the records, you need to chain the project function to apply the projection. The way you are doing ignores the projection. Your code should look like this

et id = ObjectID('5e56503cafc87b893b92827c');
let start = performance.now();
let find = mongodb.collection('libraries').find(
    {
        "$or": [
            {"organization_id": id, "type": "template"},
            {"organization_id": null,"type": "standard-template", "organizations": id}
        ]
    }).project({_id: 1}); // this is the projection

while (await find.hasNext()) {
    const doc = await find.next();
    console.log(doc); //HUGE! way more than just _id!
}

console.log(performance.now() - start); //about 12000 ms

OR

you can wrap the projection in a projection property like this

{"projection": {"_id": 1}}

either of this should work.