Search code examples
cassandrakuberneteskubectlkubernetes-helm

Last Pod In Kubernetes Stefulsets is ready after 3 times restarting


I am deploying cassandra in Kubernetes using Helm. While starting the statefulsets, For eg. with 6 pods, the last pod is starting after 3 times restart(crashloopbackoff: Backoff restarting failed container). But after 3 times restart, the pod is ready. Before that, I used PodManagementPolicy: OrderedReady. I didnt face this problem at that time. I want to start all the pods at the same time. So I give PodManagementPolicy: Parallel. Now I face this problem.


Solution

  • You can't start multiple Cassandra instances in parallell. Each Cassandra node has to bootstrap (stream data) and join the cluster. If a joining node notices another node is already joining it will crash (stop Cassandra). This is why you're getting the crashbackoff message.

    I recommend you reverse your PodManagementPolicy and setup a readinessProbe. Example script: https://github.com/instaclustr/cassandra-operator/blob/ace024626c9339650a5a76861f36af48423a35be/docker/cassandra/readiness-probe.sh