What could be the reason for spontaneous master object count drop? I've added 3 more nodes to a cluster containing 17 nodes, and suddenly there'are 1 billion records less (as reported by AMC UI)?
Replica objects goes to zero as well.
Thanks!
Partitions are going to go through state transitions. Partitions that were shifted from master to replica will no longer report those objects in the master object counts. These partitions will also be in an "acting master" state until the "desync master" partitions receive the full partition. While acting master the namespace supervisor thread (NSUP) will not count them as master or replica objects. Also the node that previously was the replica for this partition will drop the partition to allow for potential incoming migrations.
There will also be partitions where the l old master and replica nodes will both be displaced by new nodes. The old master will become an acting master until the new master receives the partition. At which point the old master will drop and the new master will begin replicating to the new replica. You will periodically observe this in AMC as the number of active tx migrations will increase.
There are other possible scenarios but the main takeaway is that once migration settles, you object counts will return to normal.