Search code examples
apache-kafkakafka-producer-api

Is there any way to ensure that duplicate records are not inserted in kafka topic?

I have been trying to implement a queuing mechanism using kafka where I want to ensure that duplicate records are not inserted into topic created.

I found that iteration is possible in consumer. Is there any way by which we can do this in producer thread as well?

Solution

  • This is known as exactly-once processing.

    You might be interested in the first part of Kafka FAQ that describes some approaches on how to avoid duplication on data production (i.e. on producer side):

    Exactly once semantics has two parts: avoiding duplication during data production and avoiding duplicates during data consumption.

    There are two approaches to getting exactly once semantics during data production:

    1. Use a single-writer per partition and every time you get a network error check the last message in that partition to see if your last write succeeded
    2. Include a primary key (UUID or something) in the message and deduplicate on the consumer.

    If you do one of these things, the log that Kafka hosts will be duplicate-free. However, reading without duplicates depends on some co-operation from the consumer too. If the consumer is periodically checkpointing its position then if it fails and restarts it will restart from the checkpointed position. Thus if the data output and the checkpoint are not written atomically it will be possible to get duplicates here as well. This problem is particular to your storage system. For example, if you are using a database you could commit these together in a transaction. The HDFS loader Camus that LinkedIn wrote does something like this for Hadoop loads. The other alternative that doesn't require a transaction is to store the offset with the data loaded and deduplicate using the topic/partition/offset combination.

    I think there are two improvements that would make this a lot easier:

    1. Producer idempotence could be done automatically and much more cheaply by optionally integrating support for this on the server.
    2. The existing high-level consumer doesn't expose a lot of the more fine grained control of offsets (e.g. to reset your position). We will be working on that soon