Apache Kafka JMX Metrics with Spring Boot

Question

All,

I have a requirement to expose the Apache Kafka metrics to the spring boot (v 2.3.0.RELEASE) actuator endpoint.

Please note, I am NOT using spring-kafka library .

I am using the following libraries

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-streams</artifactId>
            <version>2.5.0</version>
        </dependency>


        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.5.0</version>
        </dependency>

I have tried this spring.jmx.enabled=true but seems like this doesnt work. I assume this is happening because spring is not managing the kafka.

Is there a way I can bind these JMX metrics to the micrometer MeterRegistry?

I am able to make this work using the jmx-exporter provided by prometheus, but since that requires an agent running on a different port, I was hoping to make this work with default Micrometer and spring boot.

Answer 1

Have you tried KafkaClientMetrics, KafkaStreamsMetrics or KafkaConsumerMetrics? KafkaConsumerMetrics collects metrics through JMX but it is deprecated in favor of KafkaClientMetrics.

You only need to create them and bind to your registry:

KafkaConsumer<String, String> consumer;
KafkaProducer<String, String> producer;
MeterRegistry registry;

...

new KafkaClientMetrics(consumer).bindTo(registry);
new KafkaClientMetrics(producer).bindTo(registry);

Similarly for KafkaStreams:

new KafkaStreamsMetrics(kafkaStreams).bindTo(registry);

If Spring does not manage your Kafka components, why do you expect spring.jmx.enabled=true to do anything?