In our Prod cluster, I see some of the system* keyspaces with SimpleStrategy. As we are adding new DCs in our cluster we need to be on NetworkTopologyStrategy otherwise while I am running nodetool rebuild, It got failed.
I alter all keyspaces from SimpleStrategy to NetworkTopologyStrategy & rebuild works well.
During ALTER it gives warning also that you are altering system keyspace.
Another question: Opscenter Keyspace(rollup* tables) is having a hell lot of data, why I should replicate that, it should be enough to put on only 1 DC??
My question what should be the ideal strategy for system* keyspaces??
My question what should be the ideal strategy for system* keyspaces?
A little warning on this one. system and system_schema have a special replication strategy of LocalStrategy, and they should stay that way.
The other keyspaces of system_auth, system_distributed, and system_traces however, are a different story. Setting those to use NetworkTopologyStrategy with a RF 3 (assuming each DC has at least 3 nodes) for each DC is perfectly acceptable. Setting that for system_distributed, and system_traces isn't as important, but it shouldn't hurt anything.
On the other hand, system_auth should definitely be set to use NetworkTopologyStrategy with a RF specified for each DC. The reason, is that SimpleStrategy could potentially put all of its replicas in a single DC, or even zero replicas in one DC. That could cause high latency or even timeouts for auth checks, as that would result in cross-DC network traffic.
Also, if your applications use DC-specific load balancing policies, you will need to specify a RF for each DC in system_auth. As mentioned above, using SimpleStrategy could result in a DC not getting any replicas for a user. And than would prevent DC-specific applications from connecting.