Search code examples
apache-kafkaoffsetkafka-producer-apiproducer

Does Kafka provides Producer level offset?

Suppose , I've got multiple Kafka producers producing data simultaneously to a single Kafka Topic.

Is it possible to get which was the last offset produced by a given producer ?

For example :

Producers : P1 , P2

Topic : T1 with one partition Partition-0

P1 published 100 records and P2 published 150 records.

I want to find out offset of the last record published by P1 and P2 respectively.

Note that, I'm not asking for global Topic-Partition offset.

Solution

  • Kafka brokers don't track that information.

    Only the producers have that information but it's not even part of their built-in metrics. If you want to track the last offset per producer you will need to create a metric in each of your producer instances yourself.

    You can simply retrieve the offset in the RecordMetadata object returned by send() and expose it as a metric for example.