Search code examples
apache-kafkakafka-consumer-api

Kafka - Who writes to the __consumer_offsets topic?


I am a bit confused about who writes to the __consumer_offset topic.

In Kafka definitive guide ch.4 p.75 is quite generic about this detail

How does a consumer commit an offset? It produces a message to Kafka, to a special __consumer_offsets topic, with the committed offset for each partition.

In an other website I have been reading this about the Group Coordinator being an offset manager as well. I went back to the confluent documentation and I have found this article:

Remember the __consumer_offsets topic? Each consumer group stores the state of all its consumers inside a single partition of that topic. The group coordinator broker is the leader for that consumer group’s partition. This way, every broker can be a group coordinator. With enough groups, every broker will work as a coordinator for a different subset of consumer groups.

But I could not found a reference about who is writing the updates about commit offset to the __consumer_offset internal topic. Is it the consumer? Is it the group coordinator? Is it something else internal?


Solution

  • Group coordinator writes to __consumer_offsets topic. From here

    Most client libraries automatically commit offsets to Kafka for you on a periodic basis, and the responsible Kafka broker will ensure writing to the __consumer_offsets topic (therefore consumers do not write to that topic directly).