Spring Kafka - Re reading an offset after sometime

Question

I am using @KafkaListener with props as

max.poll.records to 50. (Each record takes 40-60 sec to process)

enable-auto-commit=false

ack-mode to manual immediate

Below is the logic

@KafkaListener(groupId=“ABC”, topic=“Data1” containerFactory=“myCustomContainerFactory”)
public void listen(ConsumerRecord<String, Object> record, Acknowledge ack) {

try{

process(record);

ack.acknowledge();

}

Catch(e){
 reprocess() // pause container and seek

  }
}

Other props like max.poll.interval.ms, session.timeout.ms or heartbeat are of default values

I am not able to understand whats going wrong here,

Suppose if 500 msg are published to 2 partition

1. I am not sure why the consumer is not polling records as per max.poll.records prop actually its polls all 500 msg as soon as the application starts or msg are published by producer

2. Its observed that after processing some records say approx 5-7 mins consumer re reads an offset again.. which actually was read fine processed and acknowledged..

After a hour the log file shows that same messages are read multiple times.

Any help is appreciated Thanks.

Answer 1

The default max.poll.interval.ms is 300,000 milliseconds (5 minutes).

You either need to reduce max.poll.records or increase the interval - otherwise Kafka will force a rebalance due to a non-responsive consumer.

With such a large processing time, I would recommend max.poll.records=1; you clearly don't need higher throughput.