MongoDB change stream display only selected fields

Question

I am trying to understand change stream in MongoDB.

I am trying to display only some fields of updated document.

So I did:

option={ 'full_document':'updateLookup' }
collection.watch([{"$match"  : { "operationType" : "update" }}] , **option)

which is working.

Now I want to display only some fields.

I tried:

collection.watch([{"$match"  : { "operationType" : "update" }},{"$project":{"_id":1}}] , **option)

or

collection.watch([{"$match"  : { "operationType" : "update" }}],option).aggregate({"$project":{"_id":1}})

but neither of them works.

How can I display only selected fields?

Answer 1

From the Change Event documentation page, a change stream will output this document:

{
   _id : { <BSON Object> },
   "operationType" : "<operation>",
   "fullDocument" : { <document> },
   "ns" : {
      "db" : "<database>",
      "coll" : "<collection"
   },
   "documentKey" : { "_id" : <ObjectId> },
   "updateDescription" : {
      "updatedFields" : { <document> },
      "removedFields" : [ "<field>", ... ]
   }
}

That is, that document will be the output if you put nothing inside the watch() method.

You can then use aggregation pipeline stages to filter/modify this document.

For example, if you only want to see the fields _id, a, and b from inserted/updated documents, the pipeline would be (using Python):

with db.test.watch([
    {'$project': {
        'fullDocument_id':'$fullDocument._id',
        'a':'$fullDocument.a',
        'b':'$fullDocument.b'}}], full_document='updateLookup') as stream:
    for change in stream:
        print change

Insert data into MongoDB:

> db.test.insert({a:1, b:1, c:1, d:1})

The change stream will output:

{'a': 1.0,
 'fullDocument_id': ObjectId('5aa0c2300551e941c6958f86'),
 '_id': <BSON object>,
 'b': 1.0}

Note: I projected the inserted document's _id into fullDocument_id. You can project it to _id (e.g. _id: '$fullDocument._id'), but you will lose the change stream's original _id field, which contains the resume token.

Note: I also used the parameter full_document='updateLookup' when creating the stream. Otherwise, the full document won't be shown in an update event. This is explained in the Change Event page

Note: The example above is for inserting documents, but you can easily tailor it for updates using a $match: {operationType: 'update'} stage before the $project stage.

Note: The fullDocument field that is returned on update events contain the looked up version of the document that is majority-committed to the replica set members. This may or may not be the version of the document that was modified. Any other interleaving operations between the update operation and when the change stream is returned to the client may have altered the version of the document. For example, a delete event may caused the fullDocument field to be null. See Lookup Full Document for Update Operations for more details.