HBase fully distributed mode [Zookeeper error while executing HBase shell]

Question

Following these two tutorials: i.e tutorial 1 and tutorial 2, I was able to set up HBase cluster in fully-distributed mode. Initially the cluster seems to work okay.

The 'jps' output in HMaster/ Name node

The jps output in DataNodes/ RegionServers

Nevertheless, when every I try to execute hbase shell, it seems that the HBase processors are interrupted due to some Zookeeper error. The error is pasted below:

2021-03-13 11:52:26,047 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper exists failed a│1951 HRegionServer
fter 4 attempts                                                                               │hduser@master-vm:~$ 
2021-03-13 11:52:26,048 WARN  [main] zookeeper.ZKUtil: hconnection-0x4375b0130x0, quorum=137.4│
3.49.59:2181,137.43.49.58:2181,137.43.49.50:2181,137.43.49.49:2181, baseZNode=/hbase Unable to│
 set watcher on znode (/hbase/hbaseid)                                                        │
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss│
 for /hbase/hbaseid                                                                           │
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)               │
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)               │
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)                         │
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.│
java:221)                                                                                     │
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:417)              │
        at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:6│

I tried several attempts to solve this issue (including trying out with different HBase/ Hadoop compatible versions). But still no progress.

Would like to have your input on this.

Shared below are other information required:

• in /etc/hosts file:

(I already tried commenting the HBase related hosts in /etc/hosts/, still didn'w work)

• in hbase-site.xml

Answer 1

After 5 days of hustle, I learned what went wrong. Posting my solution here. Hope it can help some of the other developers too. Also would like to thank @VV_FS for the comments.

In my scenario, I used virtual machines which I burrowed from an external party. Therefore, there were certain firewalls and other security measures. In case if you follow a similar experimental setup, these steps might help you.

To set up HBase cluster, follow the following tutorials.

1. Set up Hadoop in distributed mode.

Notes when setting up HBase in fully distributed-mode:

• Make sure to open all the ports mentioned in the post. For example, use sudo ufw allow 9000 to open port 9000. Follow the command to open all the ports in relation to running Hadoop.

2. Set up Zookeeper in distributed mode.

Notes when setting up Zookeeper in fully distributed mode:

• Make sure to open all the ports mentioned in the post. For example, use sudo ufw allow 3888 to open port 3888. Follow the command to open all the ports in relation to running Zookeeper.
• DO NOT START ZOOKEEPER NODES AFTER INSTALLATION. ZOOKEEPER WILL BE MANAGED HBASE INTERNALLY. THEREFORE, DON'T START ZOOKEEPER AT THIS STAGE.

3. Set up HBase in distributed mode.

• When setting up values for hbase-site.xml, use port number 60000 for hbase.master tag, not 60010. (thanks @VV_FS to point this out in the earlier discussion).

• Make sure to open all the ports mentioned in the post. For example, use sudo ufw allow 60000 to open port 60000. Follow the command to open all the ports in relation to running Zookeeper.

[Important thoughts]: If encounters errors, always refer to HBase logs. In my case, hbase-mater-xxxxx.log and zookeeper-master--xxx.log helped me to track down exact errors.