Spring boot Cassandra read timeout

Question

I have problem with Spring Boot + cassandra web application. It started to appear with data grown, and now it's super common scenario.

All queries sometimes doesn't work, CassandraRepository returns null. And few seconds later it works again, and next few seconds it's not working again. So web application constantly returns 200 or 404 response. The same query works in cqlsh all time.

I'm using:

• spring-boot-starter-parent#2.1.3
• spring-boot-starter-data-cassandra#2.1.3
• Cassandra 3.11.3 (multiple clusters)

Data structure:

CREATE KEYSPACE data WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '2'}  AND durable_writes = true;

CREATE TABLE data.image (
    hash text PRIMARY KEY,
    image blob
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

To configure Cassandra connection I use:

@Configuration
@EnableCassandraRepositories(basePackages = "...path...")
public class CassandraConfig extends AbstractCassandraConfiguration {

    @Bean
    public CassandraClusterFactoryBean cluster() {
        CassandraClusterFactoryBean cluster = new CassandraClusterFactoryBean();
        cluster.setContactPoints("127.0.0.1");
        cluster.setPort(91234);
        return cluster;
    }

To retrive data I'm using CassandraRepository with @Query(("select * from image where id = ?0")) QueryAnnotation. Retrieved data contains image blobs.

I think that read timeout is a problem here, servers have slow HDD disks and not so powerful CPUs. But how can I overwrite this settings with spring boot starter?

I've tried to use

SocketOptions so = new SocketOptions();
so.setConnectTimeoutMillis(10000);
so.setReadTimeoutMillis(20000);
cluster.setSocketOptions(so);

with no success.

What else can I make to have stable working solution?

Answer 1

nodetool repair helped -> after a few days everything started to work as expected. Conclusion: too much write do database.