Verify existence of message in a Kafka topic

Question

I wish to avoid sending duplicate messages to a Kafka topic.

What is the ideal way to achieve it ?

Using Java client for Apache Kafka, is there anyway to verify if a message exists before invoking KafkaProducer.send

I am referring to this doc

Answer 1

Currently (Kafka 0.10.1), there is no way to have exactly-once guarantees on write with Kafka. No matter what workaround you want to do, there will be always be a gap and you can end up with either lost messages or duplicates.

However, Kafka will add an idempotent producer (planned for 0.10.2) that will allow you to avoid duplicate writes. The target date for 0.10.2 release is beginning 2017.