Search code examples
mongodbmongodb-replica-set

Can a mongodb secondary recover if the lag is larger then the oplog?

Let's assume a secondary is fallen behind because of server crash that was not restarted within a given time. Now we have a lag of 100h but an oplog of 10h like this

db.printSlaveReplicationInfo()

source: db3
    syncedTo: TIMEINTHEPAST
    360000 secs (100.00 hrs) behind the primary
source: db1
    syncedTo: NOW
    1 secs (0 hrs) behind the primary

But then the oplog is much smaller

db.printReplicationInfo()

configured oplog size:   GIVENSIZE
log length start to end: 36000secs (10hrs)

Do I assume correctly that this secondary has no chance of catching up (RECOVERING)?

How is this situation to be resolved?

Solution

  • Answer is simple. Secondary cannot survive. In that situation you have few different possibilities to "re-initialize" that node.

    • You stop that node, empty it's data directory, restart it and let it do initial sync.

    OR

    • You stop that node, bring snapshot (or data files) of other healthy nodes data files, restart it and let it "catch" the primary.

    After you have again working replica set, resize your oplog size to much bigger. Like 15 times bigger.