Search code examples
mongodbmongodb-querymongodb-indexesflask-mongoengine

Searching strings in large mongodb collection (540GB / 4 billion documents)

I have a very large database collection of around 540GB with 4 billion items.

The item contains various metadata but one important field "message" which is free text.

I would like to be able to query it for things like:

  • the message contains "error"
  • the message contains "password"
  • the message ends with "an error occurred"

What is the best type of index to create, and how can I create it?

I was also wondering if anyone had example queries?

I having been reading into text indexes and wildcard text indexes but I'm not sure what is the best fit.

Solution

  • A text index would be a good fit here:

    db.mycoll.createIndex({ message: 'text' })

db.mycoll.find({ $text: { $search: 'error' } })