Search code examples
apache-kafkaprometheusgrafanastrimzi

Strimzi kafka exporter kafka_consumergroup_members metric


I deployed a kafka cluster with consumer and producer clients on Kubernetes using Strimzi operator. I used the following strimzi deployment file , https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/kafka-metrics.yaml

I am using Kafka exporter to monitor consumer related metrics (Messages in/out per second per topic, lag by consumer group, offsets etc..). However, I am interested in configuring Prometheus to scrape the kafka_exporter metric "kafka_consumergroup_members" for later display on Grafana. What additional configuration shall I add to the strimzi Prometheus configuration file (https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/prometheus-install/prometheus-rules.yaml) or any other deployment file (e.g., https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/kafka-metrics.yaml) so that "kafka_consumergroup_members" from the kafka_exporter metric is scraped.


Solution

  • The Kafka Exporter is a separate tool which provides additional metrics not provided by Kafka itself. It is not configurable in what metrics does it offer - you can only limit for which topics / consumer groups it will show the metrics.

    So all metrics supported by Kafka Exporter are published on its metrics endpoint and when Prometheus scapes them it should scrape all of them. So if you have the other KAfka Exporter metrics in your Prometheus, you should already have this one as well (you actuall need to have some active consumer grups for it to show up).