Search code examples
apachehadoophbasecentos7bigdata

HRegionServer shows "error telling master we are up". Showing socket exception: Invalid argument


Iam trying to create a hbase cluster in 3 centos machines. Hadoop(v - 2.8.0) is up and running on top I configured HBase(v - 1.2.5).Hbase start up is fine it started HMaster and Region servers but still it shows the follwing error in region servers and in HMaster log it shows no region servers are checked in.

2017-04-20 19:30:33,950 WARN  [regionserver/localhost/127.0.0.1:16020] regionserver.HRegionServer: error telling master we are up
com.google.protobuf.ServiceException: java.net.SocketException: Invalid argument
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:240)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
        at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2316)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:907)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Invalid argument
        at sun.nio.ch.Net.connect0(Native Method)
        at sun.nio.ch.Net.connect(Net.java:454)
        at sun.nio.ch.Net.connect(Net.java:446)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:416)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:722)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)

JPS of my master node

[hadoop@localhost bin]$ jps
20624 SecondaryNameNode
20800 ResourceManager
20401 NameNode
18061 Jps
17839 HMaster

JPS of myregion nodes are

[hadoop@localhost bin]$ jps    
11168 Jps
482 DataNode
10840 HQuorumPeer
10974 HRegionServer

hbase-site.xml of all nodes

<configuration>
<property>
        <name>hbase.master.hostname</name>
        <value>NameNode</value>
</property>
<property>
        <name>hbase.rootdir</name>
        <value>hdfs://NameNode:8020/hbase</value>
</property>
<property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
</property>
<property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>hdfs://NameNode:8020/zookeeper</value>
</property>
<property>
        <name>hbase.zookeeper.quorum</name>
        <value>DataNode1,DataNode2</value>
</property>
<property>
        <name>hbase.zookeeper.property.clientPort</name>
        <value>2181</value>
</property>
</configuration>

regionservers file contain

DataNode1
DataNode2

etc/hosts file in all nodes contain actual ips rather than loopback ips

192.168.00.00 NameNode
192.168.00.00 DataNode1
192.168.00.00 DataNode2

Note configuration is same in all nodes. Any help will be appreciated.


Solution

  • I put the following property in all region servers hbase-site.xml solved my problem.<property> <name>hbase.regionserver.hostname</name> <value>DataNode1</value> </property> <property> <name>hbase.regionserver.port</name> <value>16020</value> </property>