Monitor ksqlDB with JMX

Question

I have followed the instructions on this page: https://docs.ksqldb.io/en/latest/operate-and-deploy/monitoring/

So this is my ksqldb-server part of docker-compose:

  ksqldb-server:
    image: confluentinc/ksqldb-server:0.15.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - kafka
      - schema-registry
      - kafka-connect
    ports:
      - "8088:8088"
      - "1099:1099"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: kafka:29092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      KSQL_KSQL_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      KSQL_KSQL_CONNECT_URL: http://kafka-connect:8083
      KSQL_KSQL_QUERY_PULL_METRICS_ENABLED: "true"
      KSQL_JMX_OPTS: > 
        -Djava.rmi.server.hostname=localhost
        -Dcom.sun.management.jmxremote
        -Dcom.sun.management.jmxremote.port=1099
        -Dcom.sun.management.jmxremote.authenticate=false
        -Dcom.sun.management.jmxremote.ssl=false
        -Dcom.sun.management.jmxremote.rmi.port=1099

I have setup Prometheus in the same docker-compose file, and when I visit {prometheus-url}/targets, I see Get "http://ksqldb-server:1099/metrics": EOF

I have already tried plenty configurations during my research, including changing the -Djava.rmi.server.hostname either to the host's IP address or to ksqldb-server's container IP address, but none of them worked. Does anyone have a solution?

Answer 1

Well, six months later after having dealt with this topic once again, I managed to set this up. This follows the approach suggested by swist in my GitHub issue I created back then when I created this issue, too.

• You need JMX Exporter. Download it here

• You need a YAML file, telling the JMX exporter which metrics to export. You can get it here. If you are only interested in the ksqlDB metrics, remove all other patterns, e.g. the kafka patterns.

• Place the JMX Exporter and the YAML file on every node on which you want to monitor a ksqlDB instance

• Before starting ksqlDB, create the environment variable KSQL_JMX_OPTS as follows:

  export KSQL_JMX_OPTS="-Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.authenticate=false \ 
  -Dcom.sun.management.jmxremote.ssl=false \
  -Djava.util.logging.config.file=logging.properties \
  -javaagent:[BLUB]/jmx_prometheus_javaagent.jar=7010:ksqldb.yml"
  

  You need to either create this variable every time you have a new session or create it permantently. [BLUB] is the absolute path to your JMX JAR.

Now you can run ksqlDB and the metrics become available at port 7010 (you can specify any other free port). If you want to have a good dashboard, go with this one.