From 'Cassandra Definitive Guide':
While Cassandra provides a pluggable way to statically describe your cluster’s topology, it also provides a feature called dynamic snitching that helps optimize the routing of reads and writes over time. Here’s how it works. Your selected snitch is wrapped with another snitch called the DynamicEndpointSnitch. The dynamic snitch gets its basic understanding of the topology from the selected snitch. It then monitors the performance of requests to the other nodes, even keeping track of things like which nodes are performing compaction. The performance data is used to select the best replica for each query. This enables Cassandra to avoid routing requests to replicas that are busy or performing poorly.
Dynamic snitch monitors the nodes, in addition to info provided by configured snitch.
Doesnt it make whole snitching process slow, defeating the very purpose of snitch?
Doesnt it make whole snitching process slow, defeating the very purpose of snitch?
Yes, it absolutely does suffer from the "observer effect." Jon Haddad wrote an article a few years back (still relevant) where he covers this (point #5) and advocates for disabling it.
Disable Dynamic Snitch
Dynamic snitch is a feature that was intended to improve the performance of reads by preferring nodes which are performing better. Logically, it makes quite a bit of sense, but unfortunately it doesn’t behave that way in practice. In reality, we suffer from a bit of the Observer Effect, where the observation of the situation affects the outcome. As a result, the dynamic snitch generates quite a bit of garbage, so much in fact that using it makes everything perform significantly worse. By disabling it, we make the cluster more stable overall and end up with a net reduction in performance related problems. A fix for this is actively being worked on for Cassandra 4.0 in CASSANDRA-14459.
The full article "14 Things To Do When Setting Up a New Cassandra Cluster" can be found here: https://thelastpickle.com/blog/2019/01/30/new-cluster-recommendations.html