I am reading Kafka documentation and trying to understand the working of it. This is regarding consumers. In brief, a topic is divided into number of partitions. There are number of consumer groups, each having number of consumer instances. Now, my question is, does each partition sends sends "same" message to each consumer groups, which in turn is given to specific consumer instance within the group?
If it is, how does Kafka ensures the message is processed only by one consumer?
Kindly guide me if I am missing something.
Well to put it simply :
we have topic divided into partitions.
we have consumer that consume data from thoses topics.
Consumers are part of consumer group by sharing the same group.id.
From a topic every partitions is consumed by one consumer within a consumer groups.
Example : Topic "test" with 3 partitions. Consumer group A : with 3 consumers Consumer group B : with 2 consumers.
Ths two consumer groups A and B consumes data from the topic "test".
Within the group A every consumer (so 3) will consume one partition each whereas in group consumer B (with 2 consumer) , one consumer will read 2 partitions and the other will consume the last one. If we have a last consumer group with only one consumer inside, it will read all 3 partitions of that topic.
Hope that's help, let me know if you didn't understand.