How do we know that all the Solr nodes in a SolrCloud cluster are in sync?

I have a SolrCloud setup with 2 Solr nodes and 3 ZooKeeper nodes. shards # 1 and replicationFactor # 2.

How can I know that both the Solr nodes (leaders/replicas) are in sync and all the index data is replicated successfully?

Thanks, Amit

Solution

SolrCloud API calls are completely consistent in that when the response comes back, it has done so after ALL relevant replicas have confirmed an update received and added to the transaction log (otherwise you'll get an error).

This is worth reading about Transaction logs in particular

The indexing flow in SolrCloud is as follows:

Incoming documents are received by a node and forwarded to the proper leader.

From the leader they’re sent to all replicas for the relevant shard.

The replicas respond to their leader.

The leader responds to the originating node.

After all the leaders have responded, the originating node replies to the client. At this point, all documents have been flushed to the tlog for all the nodes in the cluster!

This is also worth reading as Solr node recovery can impact consistency: Solr consistency and recovery internals