cassandra connections spikes load issue

I am using cassandra according to the following struct: 21 nodes , AWS EC2 i3.2xlarge , version 3.11.4 .

The application is opening about 5000 connection per node (so its 100k connections per cluster) using the datastax java connection driver. Application is using autoscale and frequently opens/close connections. Number of connections to open at once by app servers can reach up to 500 per node (opens simultaneously on all nodes at once - so its 10k connections opens at the same time across the cluster)

This cause spikes of load on cassandra and cause reads and writes latency. I have noticed each time connections opens/close there are high number of reads from system_auth.roles and system_auth.role_permissions.

How can I prevent the load and resolve this issue ?

Solution

You need to modify your application to work with as small number of connections as possible. You need to have following in mind:

Create Cluster/Session object, once at start and keep it. Initialization of session is very expensive operation, it adds a load to Cassandra, and to your application as well
you may increase the number of the simultaneous requests per connection, instead of opening new connections. Protocol allows to have up to 32k requests per connection. Although, if you have too many requests in-flight, then it's a sign that your Cassandra doesn't keep with workload and can't answer fast enough. See documentation on connection pooling