Search code examples
hadoophadoop-yarnhigh-availabilitygoogle-cloud-dataproc

How to recover deleted Master Node in dataproc cluster?

My one of the Master Nodes got deleted accidentally in my dataproc cluster. Is there any way to recover that Master node or Can I spin up a new Master Node and add it to my cluster? The reason of deletion is still unknown.

Any help is really appreciated.

Solution

  • After knowing that I didn't have many options, I tried the below steps and it worked.

    1. Determine the current active NameNode(hdfs haadmin -getServiceState nn0/nn1)
    2. Create an AMI of the current active NameNode
    3. Launch a new instance from that AMI having exact same name as of deleted master-node.(This is crucial as all hdfs properties inside hdfs-site.xml are configured using this hostname only. So make sure every detail of this instance is exact same as the lost one.)

    Our AMI contains every required configuration and services, So as the new instance starts, dataproc will automatically identify the node and add it to the cluster.