I'm using a Kafka Producer and my application sends individual ProducerRecords all with the same key into a single partition, and these ProducerRecords are then batched (using batch.size and linger.ms parameters) before being sent to the brokers. I have enable.idempotence=true and acks=all.
If one record in the middle of a batch fails to be written, for example if a host crashes or a network failure or disk failure occurs or the record failed to be acked by the minimum replicas, does Kafka guarantee that all subsequent records will also not be written? Or is there a possibility that a record in the middle of a batch could be missing?
If one record in the middle of a batch fails to be written, for example if a host crashes or a network failure or disk failure occurs or the record failed to be acked by the minimum replicas, does Kafka guarantee that all subsequent records will also not be written?
Yes, if any message within a batch fails, then all messages in the same batch fail. So none of the messages within the batch will be written to the broker's disk.
Or is there a possibility that a record in the middle of a batch could be missing?
No, either all or none messages of the batch are written to the broker.
This is achieved by the separation between the Producer client thread and a local buffer that queues and batches the data before sending it physically to the broker.