I have an 7.4.0 ES cluster using ECK 1.0 and after my 3 dedicated master nodes got out of disk space, I deleted them along with the volumes to test a critical scenario.
Once the new eligible masters were created, they couldn't elect a new member. Now the cluster is stuck forever although it sees the new master eligible servers (pods in k8s).
Is there a way to force ES to elect a new master even though the previous ones are out of the picture?
Be aware that the masters had no data. All the data resides on data only nodes. Unfortunately, I cannot access them as long as a master is not elected.
Having no snapshot in place yet, I resolved myself around bringing the data nodes into a new cluster.
I recreated the 3 master nodes and made sure they joined a new cluster with an elected master. Following that, I detached the data nodes from their previous cluster and let them associate with the new one. I knew the risk and fortunately, I didn't lose data except for my xpack security credentials.
I followed the procedure here and adapted it for my ECK case by updating the init container of the statefulset.
After such pain, I cannot stress enough the value of snapshots.