Search code examples
hbasejmxambarihdp

HBase with JMX exporter BindException

I have deployed HBase on Ambari HDP. I try to take HBase metrics using JMX exporter, so I add the next configuration in 'hbase-env':

export HBASE_OPTS="$HBASE_OPTS -javaagent:/opt/jmx_exporter/jmx_exporter_javaagent.jar=7174:/opt/jmx_exporter/hbase/hbase.yml"

I run HBase without problems, but when I try to access hbase shell, it throws the next error:

Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:497)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent (InstrumentationImpl.java:386)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain (InstrumentationImpl.java:401)
Caused by: java.net.BindException: The address is already being used
at sun.nio.ch.Net.bind0 (Native Method)
at sun.nio.ch.Net.bind (Net.java:437)
at sun.nio.ch.Net.bind (Net.java:429)
at sun.nio.ch.ServerSocketChannelImpl.bind (ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind (ServerSocketAdaptor.java:74)
at sun.net.httpserver.ServerImpl.bind (ServerImpl.java:133)
at sun.net.httpserver.HttpServerImpl.bind (HttpServerImpl.java:54)
at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer. <init> (HTTPServer.java:145)
at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain (JavaAgent.java:49)
... 6 more
FATAL ERROR in native method: processing of -javaagent failed
Aborted

I already tried to change the port of the JMX exporter, but always is throwing BindException. If I delete the line of the JMX exporter in Hbase-env, it works fine and I can access to hbase shell.

Solution

  • Based in the next article: https://blog.godatadriven.com/hbase-prometheus-monitoring, I added the following to /usr/bin/hbase, so if the port is available it export there the metrics only the first time.

    if [ `lsof -n -i:7000 | grep LISTEN | wc -l` == "0" ]; then
  export HBASE_OPTS="$HBASE_OPTS -javaagent:/opt/jmx_exporter/jmx_exporter_javaagent.jar=7000:/opt/jmx_exporter/hbase/hbase.yml"
fi

    The problem was, that every time you tried to start services of HBase there were ports conflicts so it fails.