Search code examples
apache-kafkakafka-consumer-apiapache-kafka-streams

Unable to describe Kafka Streams Consumer Group


What I want to achieve is to be sure that my Kafka streams consumer does not have lag.

I have simple Kafka streams application that materialized one topic as store in form of GlobalKTable.

When I try to describe consumer on Kafka by command:

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-application-id

I can't see any results. And there is no error either. When I list all consumers by:

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --all-groups

my application consumer is listed correctly.

Any idea where to find additional information what is happening that I can't describe consumer? (Any other Kafka streams consumers that write to topics can be described correctly.)


Solution

  • If your application does only materialize a topic into a GlobalKTable no consumer group is formed. Internally, the "global consumer" does not use subscribe() but assign() and there is no consumer group.id configured (as you can verify from the logs) and no offset are committed.

    The reason is, that all application instances need to consume all topic partitions (ie, broadcast pattern). However, a consumer group is designed such that different instances read different partitions for the same topic. Also, per consumer group, only one offset can be committed per partition -- however, if multiple instance read the same partition and would commit offsets using the same group.id the commits would overwrite each other.

    Hence, using a consumer group while "broadcasting" data does not work.

    However, all consumers should expose a "lag" metrics records-lag-max and records-lag (cf https://kafka.apache.org/documentation/#consumer_fetch_monitoring). Hence, you should be able to hook in via JMX to monitor the lag. Kafka Streams includes client metrics via KafkaStreams#metrics(), too.