Kafka Producer Metrics at regular intervals using Java API

Question

I am aware of the metric() method available in Kafka API for producer metrics. I am actually trying to get the metrics at regular intervals for 10k records, 100k and so on in the below manner:

I am running my producer from Intellij IDE which will produce around a million records. I tried printing the metrics in the below manner:

for (Entry<MetricName, ? extends Metric> entry : producer.metrics().entrySet()) {
    System.out.println(entry.getKey().name() + " : " + entry.getValue().metricValue());
}

But this method is returning the overall metrics of the producer.

How can I get the interval specific request-rate & request-total producer metrics by using metrics() ?

Answer 1

You have to explicitly check the value of these metrics at the desired intervals.

Something like (simplified logic):

Producer<String, String> producer = new KafkaProducer<>(configs);

// Find the metrics you are interested in
Metric requestTotalMetric = null;
for (Entry<MetricName, ? extends Metric> entry : producer.metrics().entrySet()) {
    if ("request-total".equals(entry.getKey().name())) {
        requestTotalMetric = entry.getValue();
    }
}

for (int i = 0; i < 1000000000; i++) {
    producer.send(new ProducerRecord<String, String>("topic", "record"));
    // Get metric value at desired interval
    if (i % 100000 == 0) {
        System.out.println(i + " : " + requestTotalMetric.metricValue());
    }
}