I have a Kafka Cluster of 3 nodes with 1 ZK and 1 Broker each.
Kafka version 0.10.1.1.
Each Topic has replication-factor=3 and min.insync.replicas=2. We have checked by issuing --describe thereafter, that the Topic(s) were successfully created with correct replication-factor and ISR.
I can see after a while few of the Topics shrunk their ISR into a Single Leader.
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper zk1:2181,zk2:2181,zk3:2181 --topic topic1
Topic:topic1 PartitionCount:1 ReplicationFactor:1 Configs:
Topic: topic1 Partition: 0 Leader: 1 Replicas: 1 Isr: 1
/opt/kafka/bin/kafka-topics.sh --describe --zookeeper zk1:2181,zk2:2181,zk3:2181 --topic topic2
Topic:topic2 PartitionCount:1 ReplicationFactor:1 Configs:
Topic: topic2 Partition: 0 Leader: 1 Replicas: 1 Isr: 1
When I connect to zk Shell. I can see all Nodes are up and the controller is in broker:2
/opt/kafka/bin/zookeeper-shell.sh --describe --zookeeper zk1:2181,zk2:2181,zk3:2181
ls /brokers/ids
[0,1,2]
get /controller
{"version":1,"brokerid":2,"timestamp":"1531740996571"}
Even if I try to restart the Brokers (Only brokers or along with zookeepers), the Topics don't expand.
Any advice is highly appreciated!
Looking at the topic's descriptions, it looks like these topics were created with a replication factor of 1:
ReplicationFactor:1
Having default.replication.factor=2 and min.insync.replicas=2 set in the broker configs does not prevent the creation of topics with replication factor=1.
To fix that, you can increase the replication factor of these topics using the kafka-reassign-partitions.sh tool.
To prevent this happening, you can use the Create Topic and Alter Config Policies to reject topics with an invalid replication factor.