Search code examples
apache-kafkakafka-producer-api

If ISR is less than replication factor and producer acks are set to all, How many acks will the producer wait for?

  • RF = 3 , ISR = 3 , acks = all >>> sent successfully
  • RF = 3 , ISR = 2 , acks = all >>> sent successfully
  • RF = 3 , ISR = 1 , acks = all >>> sent successfully
  • RF = 3 , ISR = 1 , acks = all , min.isr.replicas = 3 >>> sent successfully !

So, If replication factor is 4, ISR is 3 and producer acks are set to all, How many acks will the producer wait for? I tried different scenarios, What should be the real behavior?

Solution

  • If you set acks=all broker which is the partition leader will wait all in-sync-replicas to replicate the data. In-sync-replica is a replica which is not far behind the partition leader.

    What I mean by not far behind: In Kafka when a message is sent to a topic-partition (firstly message is received and stored in leader) and if replication factor for this topic is greater than 1, then replica broker(s) send fetch request(s) to leader broker and this data is replicated to other broker(s). If replica.lag.time.max.ms is passed from last caught up, replica is considered as out-of-sync and removed from ISR list. (It is still a replica and fetch messages but leader broker doesn't wait it until catch up and became an in-sync-replica again)

    From Kafka docs:

    Configuration parameter replica.lag.time.max.ms now refers not just to the time passed since last fetch request from replica, but also to time since the replica last caught up. Replicas that are still fetching messages from leaders but did not catch up to the latest messages in replica.lag.time.max.ms will be considered out of sync.

    There is also min.insync.replicas parameter in broker config. It specifies minimum number of in-sync-replicas to continue sending message when acks=all.

    min.insync.replicas: When a producer sets acks to "all" (or "-1"), min.insync.replicas specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. If this minimum cannot be met, then the producer will raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend). When used together, min.insync.replicas and acks allow you to enforce greater durability guarantees. A typical scenario would be to create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of "all". This will ensure that the producer raises an exception if a majority of replicas do not receive a write.

    If replication factor is 4, ISR is 3 and producer acks are set to all, How many acks will the producer wait for?

    Answer: Broker which is topic-partition leader will wait 3 other brokers in ISR list to replicate the data and send acknowledgement. If number of replicas in ISR list is less than min.insync.replicas then your producer get an exception and cannot produce the data.

    Note: You can check current replica and ISR list with command below.

    bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic myTopic --describe