issue with frequent truncates in Cassandra and 24 hour ttl create large tombstones

Question

We have the below table with ttl 24 hours or 1 day. We have 4 cassandra 3.0 node cluster and there will be a spark processing on this table. Once processed, it will truncate all the data in the tables and new batch of data would be inserted. This will be a continuous process.

Problem I am seeing is , we are getting more tombstones because data is truncated frequently everyday after spark finishes processing.

If I set gc_grace_seconds to default , there will be more tombstones. If I reduce gc_grace_seconds to 1 day will it be an issue ? even if I run repair on that table every day will that be enough.

How should I approach this problem, I know frequent deletes is an antipattern in Cassandra, is there any other way to solve this issue?

TABLE b.stag (
    xxxid bigint PRIMARY KEY,
    xxxx smallint,
    xx smallint,
    xxr int,
    xxx text,
    xxx smallint,
    exxxxx smallint,
   xxxxxx tinyint,
    xxxx text,
    xxxx int,
    xxxx text,
 xxxxx text,
    xxxxx timestamp
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCom                                                                                        pactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandr                                                                                        a.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 86400
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

thank you

Answer 1

A truncate of a table should not invoke tombstones. So when you're saying "truncating" I assume you mean deleting. You can as you have already mentioned drop the gc_grace_seconds value, however this is means you have a smaller window for repairs to run to reconcile any data, make sure each node has the right tombstone for a given key etc or old data could reappear. Its a trade off.

However to be fair if you are clearing out the table each time, why not use the TRUNCATE command, this way you'll flush the table with no tombstones.