I'm trying to set up a single node cluster using Cloudera Manager, here's the latest problem I'm having.
When I try to start all services, HDFS won't start, it appears to be a problem with the Datanode - the fatal error message it gives is
Initialization failed for block pool Block pool BP-1896453393-127.0.1.1-1382183730690 (storage id DS-185043233-127.0.1.1-50010-1382183775494) service to Hadoop1/127.0.1.1:8022
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied communication with namenode: DatanodeRegistration(127.0.0.1, storageID=DS-185043233-127.0.1.1-50010-1382183775494, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=cluster25;nsid=1577346304;c=0)
at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:656)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:3495)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:899)
at
...
When I run the host inspector, it gives the following warning:
The following errors were found while checking /etc/hosts...
The hostname Hadoop1 maps to 127.0.1.1 in /etc/hosts.
This setting, typically found in Debian and Ubuntu environments, typically causes issues.
We recommend either using the standard loopback address (127.0.0.1) or specifying the appropriate IP address for your host.
From googling, it seems the problem is related to the IP address used, and the root of the problem might be in my /etc/hosts file, which looks like this.
127.0.0.1 localhost
127.0.1.1 Hadoop1
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Got this sorted, instead of using 127.0.1.1 in the /etc/hosts file I used the IP address of the machine which is in the Point-to-Point section of ifconfig.
The output of ifconfig will have a section like
ppp0 Link encap:Point-to-Point Protocol
inet addr:xxx.xxx.xxx.xxx P-t-P:xx.xx.xx.xx Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1440 Metric:1
RX packets:800950 errors:1 dropped:0 overruns:0 frame:0
TX packets:454479 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:3
RX bytes:1115444967 (1.1 GB) TX bytes:32007110 (32.0 MB)
Take the inet addr: ip from this.
Or run
ifconfig | grep P-t-P
for the exact line.