Speeding up cassandra queries if nodes are offline

Question

Working code:

using System;
using Cassandra;

    namespace CassandraSelectTest
    {
        class Program
        {
            static void Main(string[] args)
            {
                var cluster = Cluster.Builder()
                .AddContactPoints("192.168.0.18","192.168.0.21","192.168.0.22","192.168.0.23","192.168.0.24")
                .WithPort(9042)
                .Build();

                var session = cluster.Connect("test_keyspace");

                var results = session.Execute("SELECT * FROM test_table");

                foreach(var result in results)
                {
                    Console.WriteLine(result.GetValue<string>("col1"));
                }

                Console.WriteLine($"Finished");
                Console.ReadKey();
            }
        }
    }

---

Settings:

Nodes = 5
Replication = 5
Consistency = 1

---

Test:

Drop 1 node at a time to see if the above code continues to work.

---

Result:

The code continues to work after each node goes offline and then gives an error as expected when all the nodes are offline. However, the more nodes that go offline, the longer it takes for the code to run. Roughly 5 seconds extra per offline node.

---

Speeds:

5 nodes online = 1 second
4 nodes online = 5 seconds
3 nodes online = 10 seconds
2 nodes online = 15 seconds
1 node online = 20 seconds

---

Question:

Is it possible to speed up the query in the code so it does not take longer when nodes are offline? Or at least reduce the time it takes to run the query if most of the nodes are offline?

Answer 1

Are you measuring the total execution time of that code? Or are you more interested in measuring the time it takes for the application to run the queries after initialization?

During initialization (cluster.Connect), the driver iterates over the contact points collection until it is able to open the control connection successfully so if you want to make this initialization faster you must remove the offline nodes from that list.

Also by default the driver runs a warmup operation during initialization which consists of opening connections to all nodes returned by the load balancing policy so there will be a timeout here as well. You can try setting .WithPoolingOptions(new PoolingOptions().SetWarmup(false)) to make the initialization faster.

Disabling the warmup operation might make the first queries slower until the driver recognizes which nodes are offline. The recommended way to reduce the query execution time in this scenario is to use speculative retries (https://docs.datastax.com/en/developer/csharp-driver/3.12/features/speculative-retries).