We have a cassandra keyspace which has 2 tables in production. We have changed it's compression strategy from LZ4Compressor (which is default) to DeflateCompressor
using
ALTER TABLE "Keyspace"."TableName" WITH compression = {'class': 'DeflateCompressor'};
As we have around 300 GB data in each node of my cassandra 5 node cluster with replication factor 2. Is
nodetool upgradesstables recommended or not as best practice.
From all the sources that we have read
If necessary
I can use nodetool upgradesstables command. But I want to know what is actually the best practice as our data it is in production?
Sources :
When you add compression to an existing column family, existing SSTables on disk are not compressed immediately. Any new SSTables that are created will be compressed, and any existing SSTables will be compressed during the normal Cassandra compaction process. If necessary, you can force existing SSTables to be rewritten and compressed by using nodetool upgradesstables (Cassandra 1.0.4 or later) or nodetool scrub
After all nodes complete upgradesstables A large no of exceptions are being encountered in my cassandra logs
UPDATE - After running upgradesstables now my cluster is throwing a lot of errors
Sample `
ERROR [ReadRepairStage:74899] 2018-04-08 14:50:09,779 CassandraDaemon.java:229 - Exception in thread Thread[ReadRepairStage:74899,5,main] org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.service.DataResolver$RepairMergeListener.close(DataResolver.java:171) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.close(UnfilteredPartitionIterators.java:182) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:82) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.DataResolver.compareResponses(DataResolver.java:89) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:50) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.10.jar:3.10] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_144] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) ~[apache-cassandra-3.10.jar:3.10] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144] EBUG [ReadRepairStage:74889] 2018-04-08 14:50:07,777 ReadCallback.java:242 - Digest mismatch: org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(1013727261649388230, 715cb15cc5624c5a930ddfce290a690b) (d728e9a275616b0e05a0cd1b03bd9ef6 vs d41d8cd98f00b204e9800998ecf8427e) at org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233) ~[apache-cassandra-3.10.jar:3.10] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.10.jar:3.10] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144] DEBUG [GossipStage:1] 2018-04-08 14:50:08,490 FailureDetector.java:457 - Ignoring interval time of 2000213620 for /10.196.22.208 DEBUG [ReadRepairStage:74899] 2018-04-08 14:50:09,778 DataResolver.java:169 - Timeout while read-repairing after receiving all 1 data and digest responses ERROR [ReadRepairStage:74899] 2018-04-08 14:50:09,779 CassandraDaemon.java:229 - Exception in thread Thread[ReadRepairStage:74899,5,main] org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.service.DataResolver$RepairMergeListener.close(DataResolver.java:171) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.close(UnfilteredPartitionIterators.java:182) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:82) ~[apache-cassandra-3.10.jar:3.10] at org.apache.cassandra.service.DataResolver.compareResponses(DataResolver.java:89) ~[apache-cassandra-3.10.jar:3.10]`
When you use nodetool upgradesstables it writes new SSTables from existing but using the new options that you specified. This is IO-intensive process that may affect performance of your cluster, so you need to plan it accordingly. You also need to have enough disk space to perform this operation. This command should also run as the same user that runs Cassandra.
It's really depends on your needs - if it's not urgent, you can simply wait until the normal compaction occurs, and then data will be re-compressed.