When there's a failure writing a message to all ISR replicas, but the message is already persisted on the leader, what happens?
Even if the request fails, is the data still is available for consumers?
Are consumers able to read "uncommitted" data?
The message is not visible to the KafkaConsumer until the topic configuration min.insync.replicas
is fulfilled.
Kafka is expecting the failed replica broker to get up and running again, so the replication can complete.
Note, that this scenario is only relevant if your KafkaProducer configuration acks
is set to all
. When you set it to 0 or 1, the Consumer will be able to consume the message as soon as the data arrives at the partition leader.
In general, the client (here KafkaProducer) only communicates with the partition leader and depending on its mode such as synchronous, asynchronous or fire-and-forget and its acks configuration waits or does not wait for a reply. The replication of the data itself is independent of the producer and is handled by a (single-thread) on the partition leader broker. The leader will only notify on the success or failure on the replication and the leader/replicas continue to ensure to satisfy the replication set with the topic configuration replication.factor
.
The producer may still try to resend the message in case the message was only acknowledged by producer but not by the replicas, depending on its acks setting and if the exception is retriable or not. In that case you will end up having duplicate values. You can avoid this by enabling idempotence.