Cassandra: how to reduce the number of tombstones in a table? (tombstone_compaction_interval, gc_grace_seconds and LevelledCompactionStrategy)

Question

I've a table where I insert data with a TTL of 1 minute and I have a warning in DSE OpsCenter about the high number of tombstones in that table. Which does make sense since in average 80 records per minute are inserted in this table. So for example for a full day 80 * 60 * 24 = 115200 records inserted and TTL'ed in one day.

My question is what should I do in order to decrease the number of tombstones in this table?

I've been been looking into tombstone_compaction_interval and gc_grace_seconds and this is where it gets a bit confusing as I'm having problems to understand the exact impact of these properties on the tombstones (even after reading the documentation provided by DataStax - http://docs.datastax.com/en/cql/3.1/cql/cql_reference/compactSubprop.html and http://docs.datastax.com/en/cql/3.1/cql/cql_reference/tabProp.html).

I've also been looking into LevelledCompactionStrategy (https://www.datastax.com/dev/blog/leveled-compaction-in-apache-cassandra) since it also does seem to impact the tombstones compaction although I don't fully understand why.

So I'm hoping someone will be able to help me better understand how this all works, or even just let me know if I'm going in the right direction.

Answer 1

Please read this http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html. Very good read.

Overall: gc_grace_seconds parameter is the minimal time that tombstones will be kept on disk after data has been deleted. We need to make sure that all the replicas received the delete and have all tombstones stored to avoid having zombie data issues. By default its 10 days.

tombstone_compaction_interval: As part of this JIRA (https://issues.apache.org/jira/browse/CASSANDRA-4781), this property got introduced. When the compaction ratio was high enough to trigger a single-SSTable compaction, but that tombstones were not evicted due to overlapping SSTables.

I am not sure about your current datamodel but here are my suggestions.

1. Probably you have to change your DM. Please read https://academy.datastax.com/resources/getting-started-time-series-data-modeling and Time series modelling( with start & end date) in cassandra
2. Change write pattern.
3. Change read pattern. Try to read only active data. (As per your current DM, when you are reading it, its going through tombstone cells in-order to reach active cells)
4. Try to use TimeWindowCompactionStrategy and tune it as per your workload. (http://thelastpickle.com/blog/2017/01/10/twcs-part2.html)
5. If you are use TTL while inserting (like with INSERT or UPDATE stmnt), see if you can change it to the Table level.

If you are using STCS and want to change compaction sub-properties, probably you could change unchecked_tombstone_compaction=true and min_threshold=3 (little bit aggressive)