We are using cassandra 2.1.14. Currently large partition warning is seen on system.hints table.
How to make sure that system.hints table doesn't have wide partitions ? Note that we don't want to upgrade to cassandra 3 now.
Is there a periodic way to clean up system.hints ?
Will this cause I/O spike in cassandra cluster ?
Log:
Compacting large partition system/hints:
10ad72eb-0240-4b94-b73e-eb4dc2aa759a (481568345 bytes)
How to make sure that system.hints table doesn't have wide partitions?
There's not really much you can do about that. system.hints is partitioned on target_id which is the hostID of the target node. If 10000 hints build up for one node, there's really no other place for them to go.
Is there a periodic way to clean up system.hints?
As mentioned above, hints should TTL after 3 hours. This failsafe is meant to keep the system.hints table from getting too out of control. But it's not at all fool-proof.
One way to be sure, would be to clear them out via nodetool:
nodetool truncatehints
Will this cause I/O spike in cassandra cluster?
Running nodetool truncatehints is fairly innocuous. I haven't noticed a spike from running it before.