I am using a 3 broker kafka. If I stop the third node: no problem, I can consume LOGS from node 1 or 2. If I stop the second node : no problem, I can consume LOGS from node 1 or 3. If I stop the first node : I am not able to consume anything.
When I restart kafka-runner service . Logs are here and was wrote properly.
my topic configuration is: Topic: log_topic TopicId: xxx PartitionCount: 32 ReplicationFactor: 3 Configs: segment.bytes=1073741824,retention.ms=7200000 Topic: log_topic Partition: 0 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1 Topic: log_topic Partition: 1 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1 Topic: log_topic Partition: 2 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1 Topic: log_topic Partition: 3 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1 Topic: log_topic Partition: 4 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1 Topic: log_topic Partition: 5 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1 Topic: log_topic Partition: 6 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1 Topic: log_topic Partition: 7 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1 Topic: log_topic Partition: 8 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1 Topic: log_topic Partition: 9 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1 Topic: log_topic Partition: 10 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1 Topic: log_topic Partition: 11 Leader: 3 Replicas: 3,1,2 Isr: 2,3,1 Topic: log_topic Partition: 12 Leader: 3 Replicas: 1,3,2 Isr: 2,3,1 Topic: log_topic Partition: 13 Leader: 2 Replicas: 2,1,3 Isr: 2,3,1 Topic: log_topic Partition: 14 Leader: 3 Replicas: 3,2,1 Isr: 2,3,1 Topic: log_topic Partition: 15 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1 Topic: log_topic Partition: 16 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1 Topic: log_topic Partition: 17 Leader: 3 Replicas: 3,1,2 Isr: 2,3,1 Topic: log_topic Partition: 18 Leader: 3 Replicas: 1,3,2 Isr: 2,3,1 Topic: log_topic Partition: 19 Leader: 2 Replicas: 2,1,3 Isr: 2,3,1 Topic: log_topic Partition: 20 Leader: 3 Replicas: 3,2,1 Isr: 2,3,1 Topic: log_topic Partition: 21 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1 Topic: log_topic Partition: 22 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1 Topic: log_topic Partition: 23 Leader: 3 Replicas: 3,1,2 Isr: 2,3,1 Topic: log_topic Partition: 24 Leader: 3 Replicas: 1,3,2 Isr: 2,3,1 Topic: log_topic Partition: 25 Leader: 2 Replicas: 2,1,3 Isr: 2,3,1 Topic: log_topic Partition: 26 Leader: 3 Replicas: 3,2,1 Isr: 2,3,1 Topic: log_topic Partition: 27 Leader: 2 Replicas: 1,2,3 Isr: 2,3,1 Topic: log_topic Partition: 28 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1 Topic: log_topic Partition: 29 Leader: 3 Replicas: 3,1,2 Isr: 2,3,1 Topic: log_topic Partition: 30 Leader: 3 Replicas: 1,3,2 Isr: 2,3,1 Topic: log_topic Partition: 31 Leader: 2 Replicas: 2,1,3 Isr: 2,3,1
When I stop leader node. leader changes properly.
Any Idea?
This seems like a configuration issue with the __consumer_offsets topic.
_consumer_offsets is used to store information about committed offsets for each topic:partition per group of consumers (groupID)
Whenever a consumer in a group reads data from kafka, it periodically stores the information on the point till the data is read in a special topic called __consumer_offsets. This is done so that the consumer can resume from its last position in case of failure.
Issue
The replication factor for __consumer_offsets topic would have been set to 1 and all the partitions for the topic would also be present in broker 1.
You can describe the __consumer_offsets topic to verify above. If this is the case, then the easiest way to resolve the issue would be to delete the __consumer_offsets topic and recreate it with the desired replication factor.
If deletion is not an option, then following the instructions in the thread below