What may cause huge load in Kafka `__consumer_offsets` topic?

Question

I have simple observation in my Kafka cluster (Kafka 0.11.0.0).

According to jmx information the __consumer_offsets topic constantly is loaded with 10 times more messages than sum of all messages in all other topics. I had also conected console consumer to this topic and I can measure similar values.

• What could be the reason?
• How can I check what is Kafka broker doing and generate such a load on its own?

Answer 1

to read __consumer_offsets topic:

bin/kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server brokers --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" --new-consumer --consumer.config consumer.conf

for kafka 11 use formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter"

where consumer.conf has one line

exclude.internal.topics=false