I have installed hbase view brew on mac. It has been some time since the last time I started the hbase cluster: and predictably zookeeper does not work.
2014-12-02 18:56:40,250 DEBUG [main-EventThread] master.ActiveMasterManager: A master is now available
2014-12-02 18:56:40,250 WARN [M:0;192.168.255.170:51072] hbase.ZNodeClearer: Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!)
2014-12-02 18:56:40,250 INFO [M:0;192.168.255.170:51072] master.ActiveMasterManager: Registered Active Master=192.168.255.170,51072,1417575388387
2014-12-02 18:56:40,256 INFO [M:0;192.168.255.170:51072] Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
2014-12-02 18:56:40,289 INFO [main] regionserver.ShutdownHook: Installed shutdown hook thread: Shutdownhook:RS:0;192.168.255.170:51093
2014-12-02 18:56:40,292 INFO [RS:0;192.168.255.170:51093] zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=regionserver:51093, quorum=localhost:2181, baseZNode=/hbase
2014-12-02 18:56:40,293 INFO [RS:0;192.168.255.170:51093] zookeeper.RecoverableZooKeeper: Process identifier=regionserver:51093 connecting to ZooKeeper ensemble=localhost:2181
2014-12-02 18:56:40,293 INFO [RS:0;192.168.255.170:51093-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-12-02 18:56:40,294 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /0:0:0:0:0:0:0:1:51098
2014-12-02 18:56:40,294 INFO [RS:0;192.168.255.170:51093-SendThread(localhost:2181)] zookeeper.ClientCnxn: Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
2014-12-02 18:56:40,295 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /0:0:0:0:0:0:0:1:51098
2014-12-02 18:56:40,295 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x14a0e1397700001 with negotiated timeout 40000 for client /0:0:0:0:0:0:0:1:51098
2014-12-02 18:56:40,295 INFO [RS:0;192.168.255.170:51093-SendThread(localhost:2181)] zookeeper.ClientCnxn: Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a0e1397700001, negotiated timeout = 40000
2014-12-02 18:56:50,343 FATAL [M:0;192.168.255.170:51072] master.HMaster: Unhandled exception. Starting shutdown.
java.net.ConnectException: Call From mellyrn.local/192.168.255.170 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy12.setSafeMode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy12.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode(ClientNamenodeProtocolTranslatorPB.java:561)
at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2146)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:983)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:967)
at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:433)
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:850)
at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:436)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:147)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:789)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:606)
at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.run(HMasterCommandLine.java:261)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)
at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
at org.apache.hadoop.ipc.Client.call(Client.java:1318)
... 23 more
2014-12-02 18:56:50,346 INFO [M:0;192.168.255.170:51072] master.HMaster: Aborting
2014-12-02 18:56:50,346 DEBUG [M:0;192.168.255.170:51072] master.HMaster: Stopping service threads
2014-12-02 18:56:50,347 INFO [M:0;192.168.255.170:51072] ipc.RpcServer: Stopping server on 51072
2014-12-02 18:56:50,347 INFO [RpcServer.listener,port=51072] ipc.RpcServer: RpcServer.listener,port=51072: stopping
2014-12-02 18:56:50,348 INFO [M:0;192.168.255.170:51072] master.HMaster: Stopping infoServer
2014-12-02 18:56:50,348 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2014-12-02 18:56:50,348 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2014-12-02 18:56:50,350 INFO [M:0;192.168.255.170:51072] mortbay.log: Stopped [email protected]:60010
2014-12-02 18:56:50,465 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x14a0e1397700000
2014-12-02 18:56:50,465 INFO [M:0;192.168.255.170:51072] zookeeper.ZooKeeper: Session: 0x14a0e1397700000 closed
2014-12-02 18:56:50,465 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
Any ideas how to start this cluster back up?
Make sure you explicitly set the properties for your interface lo0 in the hbase-site.xml file:
<property>
<name>hbase.zookeeper.dns.interface</name>
<value>lo0</value>
</property>
<property>
<name>hbase.regionserver.dns.interface</name>
<value>lo0</value>
</property>
<property>
<name>hbase.master.dns.interface</name>
<value>lo0</value>
</property>
I found that when my wifi was on, if these entries were missing, zookeeper filed to start.