Search code examples
javajmxnewrelic

Unable to get JMX monitoring data in new relic

We are trying to monitor the idle objects count in JedisPool ( internally uses Apache Generic-Pool API supporting JMX).

We added appropriate configuration in extension newrelic file based on https://docs.newrelic.com/docs/apm/agents/java-agent/custom-instrumentation/java-agent-custom-jmx-instrumentation-yaml/ .

We are able to get appropriate JMX Metric log in newrelic_agent.log present in our server as follows:

2023-05-22T09:09:10,940+0000 [1 1] com.newrelic.agent.extension.ExtensionService FINER: Reading custom extension file /usr/local/tomcat/newrelic/extensions/GenericObjectPool.yml 
2023-05-22T09:09:10,943+0000 [1 1] com.newrelic.agent.extension.ExtensionService FINER: Adding extension with name GenericObjectPool and version 1.0 2023-05-22T09:09:10,950+0000 [1 1] com.newrelic FINER: Adding JMX config: object_name: org.apache.commons.pool2:type=GenericObjectPool,name=* attributes: [MinIdle type: simple, MaxIdle type: simple, MaxTotal type: simple, NumActive type: simple, NumIdle type: simple, NumWaiters type: simple] 
2023-05-22T09:10:36,982+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: JMX Service : MBeans query java.lang:type=ClassLoading, matches 1 2023-05-22T09:10:36,987+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: Recording JMX metric LoadedClassCount : 26,825 
2023-05-22T09:10:36,987+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: Recording JMX metric UnloadedClassCount : 1 
2023-05-22T09:10:36,987+0000 [1 146] com.newrelic FINER: JMX Metric: JmxBuiltIn/Classes/Loaded Value: 26,825 
2023-05-22T09:10:36,987+0000 [1 146] com.newrelic FINER: JMX Metric: JmxBuiltIn/Classes/Unloaded Value: 1 
2023-05-22T09:10:36,987+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: JMX Service : MBeans query java.lang:type=Threading, matches 1 
2023-05-22T09:10:36,988+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: Recording JMX metric ThreadCount : 95 
2023-05-22T09:10:36,988+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: Recording JMX metric TotalStartedThreadCount : 578 
2023-05-22T09:10:36,988+0000 [1 146] com.newrelic FINER: JMX Metric: JmxBuiltIn/Threads/Thread Count Value: 95 
2023-05-22T09:10:36,988+0000 [1 146] com.newrelic FINER: JMX Metric: JmxBuiltIn/Threads/TotalStartedCount Value: 578 
2023-05-22T09:10:36,989+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: JMX Service : MBeans query org.apache.commons.pool2:type=GenericObjectPool,name=*, matches 12 2023-05-22T09:10:36,989+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: Recording JMX metric MaxTotal : 5 
2023-05-22T09:10:36,989+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: Recording JMX metric NumActive : 0 
2023-05-22T09:10:36,989+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: Recording JMX metric NumWaiters : 0 
2023-05-22T09:10:36,989+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: Recording JMX metric NumIdle : 5 
2023-05-22T09:10:36,989+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: Recording JMX metric MinIdle : 5 
2023-05-22T09:10:36,989+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: Recording JMX metric MaxIdle : 5 
2023-05-22T09:10:36,989+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: Recording JMX metric MaxTotal : 5 
2023-05-22T09:10:36,989+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: Recording JMX metric NumActive : 0 
2023-05-22T09:10:36,989+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: Recording JMX metric NumWaiters : 0 
2023-05-22T09:10:36,989+0000 [1 146] com.newrelic.agent.jmx.JmxService FINER: Recording JMX metric NumIdle : 5 

But we are unable to find same JMX Metric data anywhere in newrelic dashboard or tables.

Can you please help in knowing how can we see JMX Metric data in new relic.

I asked same question at https://forum.newrelic.com/s/hubtopic/aAX8W0000015AasWAE/unable-to-get-jmx-monitoring-data-in-new-relic?t=1684835773973 .

Solution

  • According to the documentation it will show up in Metrics Explorer.

    Metrics explorer

    The link to Metrics explorer is inside your APM page, at the bottom of the 2nd menu on the left. Then you can filter using jmx/org and it should show the metrics.

    Below is further troubleshooting information, which are not helpful to you since you can see in the logs that the data is being properly collected, but may help others that look into this question in the future.

    If it does not show there, take a look at your agent logs to see if your extension is loading correctly.

    An error message would look like:

    com.newrelic.agent.extension.ExtensionService ERROR: Unable to parse extension. Check permissions on /path/to/extensions/ext.yml.  {error message}

    Also search for messages for the query and see if there are any matches. These log messages look like:

    com.newrelic FINER: JMX Service : MBeans query org.apache.commons.pool2:type=GenericObjectPool,name=*, matches 1

    If there are no matches, there maybe something wrong with the query in your extension. You may need to increase your log level to FINER to see these lines.