Monitoring JVMs Remotely in a Production Environment

Question

I have used jConsole, to monitor JVMs through its GUI, and sometimes to save performance data to csv by simply right clicking on graphs. But now I have a requirement of writing JVM performance data such as Heap Memory Usage and CPU Usage of remote JVMs directly into a local file (with timestamp) without using the GUI.

I have tried many tools including the followings but they don't have the required functionalities such as connecting to remote JVMs and writing outputs to local files.

1. JVM Top
2. JVM Mon

Can anyone suggest me a tool that can be used to get csv outputs which are similar to the results that can be generated through jConsole, but without using GUI.

Answer 1

I normally use cmdline-jmxclient to pull my MBean data, I use influx to store the data and later show it on Grafana for dashboarding, below is an example shell script which invokes the jmxclient

cmdline-jmxclient:

If you have an MBean: com.yourcompany.data:type=datasystem,id=0 With an Operation called: jmxRefresh()

Then you can write a simple bash script (assuming you download cmdline-jmxclient-0.10.3.jar and put in the same directory as your script):

#!/bin/bash

cmdLineJMXJar=./cmdline-jmxclient-0.10.3.jar
user=yourUser
password=yourPassword
jmxHost=localhost
port=9003

#No User and password so pass '-'
echo "Available Operations for com.yourcompany.data:type=datasystem,id=0"
java -jar ${cmdLineJMXJar} ${user}:${password} ${jmxHost}:${port} com.yourcompany.data:type=datasystem,id=0

echo "Executing XML update..."
java -jar ${cmdLineJMXJar} - ${jmxHost}:${port} com.yourcompany.data:type=datasystem,id=0 jmxRefresh

Similarly, you can get the heap memory or any other MBean's exposed in your JVM.

Another sample script is as below

#!/bin/bash

if [ $# -ne 3 ]; then
    echo "You need to supply both JMX host and port and metric. Exiting ..."
    exit 1
fi

JAVA_BIN="/usr/bin/java"
JMX_CMDLINE="./cmdline-jmxclient-0.10.3.jar"
APP_NAME="${1}"

if [ $3 = "DaemonThreadCount" -o $3 = "all" ]; then

  TEMPFILE=`mktemp`
  $JAVA_BIN -jar $JMX_CMDLINE - ${1}:${2}  "java.lang:type=Threading" DaemonThreadCount 2>> $TEMPFILE
  VALUE=`grep DaemonThreadCount $TEMPFILE | sed "s/.*DaemonThreadCount: //g" | cut -f2 -d:` 
  echo "$VALUE"
  rm -f $TEMPFILE

fi

if [ $3 = "HeapMemoryUsage" -o $3 = "all" ]; then

  TEMPFILE=`mktemp`
  $JAVA_BIN -jar $JMX_CMDLINE - ${1}:${2}  "java.lang:type=Memory" HeapMemoryUsage 2>> $TEMPFILE

  OUTPUT=`cat $TEMPFILE | egrep "committed:|used:" | sed "s/: /=/g"`

  for line in $OUTPUT
  do
    NAME=`echo $line | cut -f1 -d=`
    VALUE=`echo $line | cut -f2 -d=`
  done

  cat $TEMPFILE | grep used | cut -f2 -d" "
  rm -f $TEMPFILE

fi


if [ $3 = "PSPermGen" -o $3 = "all" ]; then

  rm -f $TEMPFILE

  $JAVA_BIN -jar $JMX_CMDLINE - ${1}:${2}  "java.lang:name=PS Perm Gen,type=MemoryPool" Usage 2>> $TEMPFILE

  OUTPUT=`cat $TEMPFILE | egrep "committed:|used:" | sed "s/: /=/g"`

  for line in $OUTPUT
  do
    NAME=`echo $line | cut -f1 -d=`
    VALUE=`echo $line | cut -f2 -d=`
    echo "$VALUE"
  done

fi