MongoDB Write Issue: Different DBs showing different count for same documents

Question

I am trying to BulkInsert certain documents in different collections in 2 Databases in MongoDB.

MongoClient mongoClient2 = this.getMongoClient();
MongoDatabase currentDB = mongoClient2.getDatabase(splits[0]);
MongoCollection<Document> currentCollectionNew = currentDB.getCollection(splits[1])
                        .withWriteConcern(WriteConcern.MAJORITY.withJournal(true))
                        .withReadConcern(ReadConcern.MAJORITY);

BulkWriteOptions bulkWriteOptions = new BulkWriteOptions();
bulkWriteOptions.ordered(true);
bulkWriteOptions.bypassDocumentValidation(true);
try {
     BulkWriteResult bulkWriteResult = urrentCollectionNew.bulkWrite(listDoc,
                            bulkWriteOptions);

   logger.info("bulkWriteResult inserted count in MAIN: " + bulkWriteResult.getInsertedCount());
   logger.info("bulkWriteResult modified count  in MAIN: " + bulkWriteResult.getModifiedCount());
   logger.info("bulkWriteResult matched count in MAIN : " + bulkWriteResult.getMatchedCount());
   logger.info("bulkWriteResult deleted count in MAIN : " + bulkWriteResult.getDeletedCount());
    logger.info("bulkWriteResult upserted count in MAIN : " + bulkWriteResult.getUpserts().size());
    logger.info("bulkWriteResult was acknowledged in MAIN : " + bulkWriteResult.wasAcknowledged());

    mongoClient2.close()
 } catch (Exception e) {
    logger.warn("Error in bulkWriting main DB: {} ", e.getMessage());
    logger.error(e.getMessage(), e);
                }

    MongoCollection<Document> mongoStageCollection = objFactory.getCollectionObject(resourceType, true);
    String[] splitsStage = mongoStageCollection.getNamespace().getFullName().split("\\.");
    MongoClient mongoClient3 = this.getMongoClient();
    MongoDatabase newStageDB = mongoClient3.getDatabase(splitsStage[0]);
    MongoCollection<Document> stageCollectionNew = newStageDB.getCollection(splitsStage[1])
                        .withWriteConcern(WriteConcern.MAJORITY.withJournal(true))
    .withReadConcern(ReadConcern.MAJORITY);

    logger.info("mongoStageCollection.getWriteConcern(): {} ", mongoStageCollection.getWriteConcern());
    logger.info("mongoStageCollection.getReadConcern(): {} ",
      mongoStageCollection.getReadConcern().toString());
    logger.info("mongoStageCollection.getReadPreference(): {}",
                        mongoStageCollection.getReadPreference().getName());

    try {
          BulkWriteResult bulkWriteResult = stageCollectionNew.bulkWrite(listDoc, bulkWriteOptions);

          logger.info("bulkWriteResult inserted count in STAGING: " + bulkWriteResult.getInsertedCount());
          logger.info("bulkWriteResult modified count  in STAGING: " + bulkWriteResult.getModifiedCount());
           logger.info("bulkWriteResult matched count in STAGING: " + bulkWriteResult.getMatchedCount());
           logger.info("bulkWriteResult deleted count in STAGING: " + bulkWriteResult.getDeletedCount());
           logger.info("bulkWriteResult upserted count in STAGING: " + bulkWriteResult.getUpserts().size());
           logger.info("bulkWriteResult was acknowledged in STAGING: " + bulkWriteResult.wasAcknowledged());
          mongoClient3.close();
                } catch (Exception e) {
                    logger.warn("Error in bulkWriting STAGING DB: {} ", e.getMessage());
                    logger.error(e.getMessage(), e);
                }

e.g. 2 DBs are FHIR and FHIR_Stage. Identical Collections are created inside both the DBs. FHIR.Condition and FHIR_STAGE.Condition

FHIR.Observation and FHIR_STAGE.Observation

and so on...

FHIR is supposed to have all the data while FHIR_Stage is supposed to have only the incremental data. However, in the initial load both the DBs should contain exactly the same data.

What I am seeing is that the counts in the Collections in these 2 DBs does not match i.e. count in FHIR.Condition is not same as in FHIR_STAGE.Condition

The issue here is this mismatch occurs randomly, i.e. it matches sometimes and sometimes it doesn't ( when I cleanup everything and re-run the initial load) And this happens for different Collections in those 2 DBs. And there is NO pattern to it, randomly some collection count wont match and sometimes everything would match.

I have not be able to wrap my head around this for a week now. Any help is greatly appreciated.

MongoDB Setup:

We have a 3 node (VM) cluster. We have 3 shards running, each shard is a 3 member replica set. Each node is a primary for one of the replica set.

The cluster is secured using x509 certificates.

I see no errors either in sh.status() or rs.status(). There is also NO replication lag.

The DBs and the Collections are created dynamically from the Java code depending on some business logic. And I am also enabling sharding on the DB and then on the collections from the code.

WriteConcern - Majority

ReadPreference - Primary

ReadConcern - Majority

Mongo-version : 3.4.15 Mongo Java Driver : 3.4.2

FYI -- the same code base works as expected on a standalone MongoDB.

Thanks in anticipation.

I will be happy to share more information if needed.

P.S.

If it makes any difference, the process writing to MongoDB is a Kafka Consumer

Answer 1

We found the issue a few days after posting this -

Ours is sharded MongoDB cluster.

This was because we weren't using the count() function from the aggregation pipeline instead relying on db.collectionName.count()

Moving to aggregation pipeline, we could see equal documents in both DBs.

Reference points it out.