How can I expand a PVC for Cassandra on AKS without losing data?

Question

I need to start by saying that I have no experience using Cassandra and I am not the one who who created this deployment.

I have Cassandra running in a cluster in AKS. The PVC as configured in the statefulset is 1000Gi. Currently the pods are out of storage and are in a constant unhealthy state.

I am looking to expand the volumes attached to the pods. The problem I am facing is that I cannot scale down the statefulset because the statefulsets only scale down when all their pods are healthy.

I even tried deleting the statefulset and then recreateing it with a larger PVC (as recomended here)

Howerver, I can't seem to delete the statefulset. It looks to me like the CassandraDatacenter CRD keeps recreating the statefulset as soon as I delete it. Giving me no time to change anything.

My question are as follows:

1. Is there a standard way to expand the volume without losing data?
2. What would happen if I scale down the replicas in the CassandraDatacenter? Will it delete the PVC or keep it?
3. If there is no standard, does anyone have any ideas on how to accomplish expanding the volume size without losing storage?

Answer 1

Ordinarily in a Cassandra cluster, the best practice is to scale horizontally (not vertically). You want more Cassandra nodes to spread the load out to achieve maximum throughput.

The equivalent in Kubernetes is to scale up your deployment. As you increase the node count, the amount of data on each individual Cassandra node will decrease proportionally.

If you really want to resize the PVC, you will only be able to do it dynamically if you have enabled allowVolumeExpansion. You won't lose data as you do this.

Deleting a STS isn't going to work because by design it will be automatically replaced as you already know. You also won't be able to scale down because there isn't enough capacity (disk space) in your cluster if you do. Cheers!