Hi I am working on kafka. I am trying to understand basics of kafka. I am learning kafka now. I installed kafka using docker. Currently I have one broker. I created topic with 3 partitions using below command.
kafka-topics --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 3 --topic topic2
After that I created producer as below.
kafka-console-producer --broker-list localhost:9092 --topic topic2
>This is my producer
I am totally confused here. When I add above data, my data sits in three partitions or one partition? because above I created three partitions. In partition we have offset starting with zero. So in the above example, When I enter This is my producer whole text will sit at offset 0 or one character sit in one offset? This is very basic I know but none of the documentation talks about this!
Next coming to consumer part, If I want to consume some data, If data sitting in different partition, How data will come from different partitions or how data consolidation will happen? Can someone help me to understand basics? Any help would be greatly appreciated. Thanks
Each message will be assigned to a different partition in a round-robin fashion. However, messages with the same key will be inserted to the same partition.
If you have N partitions, then you can have up to N consumers within the same consumer group each of which reading from a single partition. When you have less consumers than partitions, then some of the consumers will read from more than one partition. Also, if you have more consumers than partitions then some of the consumers will be inactive and will receive no messages at all.