Search code examples
pythonmongodbpymongomemory-limit

How to Update a large mongodb collection?

I am using pymongo in python to update a large collection in mongodb. I want to prevent adding duplicate objects in the collection. So, I used this command to update entire mongodb collection:

mycollection.update_one({'obj_id': "MY_OBJ_ID"}, {"$set": {"my_object": obj }}, upsert=True)

When the collecion size grow up, this error is shown:

Resulting document after update is larger than 16777216, full error: {'index': 0, 'code': 17419, 'errmsg': 'Resulting document after update is larger than 16777216'}

the question is, How Can I Update this Collection? I am new in mongodb

Solution

  • The error has nothing to do with memory. MongoDB limits the size of any document in a collection to 16Mb. Your update that is adding the obj item would mean the resulting document is over 16Mb.

    You will need to structure your data so you keep below this limit. Having all your data in one document is not the recommended approach. You should consider using multiple documents in the collection.