Search code examples
mongodbetlmongorestore

Are mongorestore actions adding to oplog?


When I perform a command like:

mongorestore --drop --db my_db dump/my_db

Are deletion and addition to database stored in oplog events so ETL tools like Fivetran, Hevo, etc are able to catch these events and perform the same operations on a target data warehouse?


Solution

  • Assuming the destination for your mongorestore command is a replica set or sharded cluster, all data changes have to be replicated via the oplog. Applications can watch a stream of events using the Change Streams API (MongoDB 3.6+) or by tailing the oplog directly. The oplog format is internal and subject to change between versions of MongoDB, so integration based on the Change Streams API (which builds on the oplog) is recommended.

    The Change Streams API allows monitoring for changes at a collection level in MongoDB 3.6 as well as scoped to a database or deployment in MongoDB 4.0+. You can have multiple Change Streams as needed to support your ETL use cases. Change streams can also modify their output which may be useful for redaction or limiting the data sent to separate ETL processes.