Kafka producer losing message even if set acks=all

Question

This is my config:

props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, "1");
props.put(ProducerConfig.LINGER_MS_CONFIG, "1");

and

try {
    producer.send(record);
} catch (Throwable ex) {
    log.error(ex, "exception.");
}

but we found message missing.

if the network dithering will lead this?

do we need to send with Callback?

producer.send(record, new Callback() {
  @Override
  public void onCompletion(RecordMetadata metadata, Exception exception) {}
})

Answer 1

A Kafka Producer has basically three different modes to produce messages to Kafka:

• fire-and-forget
• synchronous
• asynchronous

When only calling producer.send(record) you are applying the fire-and-forget mode. The configuration acks only ensures that the message will be seen as a successful write to Kafka if all replications have acknowledged that message. However, your producer does not wait for the reply.

As you have mentioned, you could make use of a callback to understand the reply from the broker. This will be the asychronous mode. But do not forget to also flush() the buffered records.

Another option would be to apply the synchronous which can be achieved by a blocking method get that waits for the ack-response from the broker. You only have to change the only line of code to

producer.send(record).get();

Edit: It might be worth increasing the retries to a number larger than 1. The Producer Callback can return retrieable exceptions which could be solved with more than one retry. To understand all callback exceptions you can have a look at another SO question on Producer Callback.