How a kafka consumer with @KafkaListener annotation handles max.poll.records

Question

I'm using spring boot 2.1.7.RELEASE and spring-kafka 2.2.7.RELEASE.And I'm using @KafkaListener annotation to create a consumer and I'm using all default settings for the consumer.

As per the apache kafka documentation, the default value for 'max.poll.records' is 500.

Here I'm trying to understand, how spring is handling the records processing. Now my question is, If we have already published 500 messages onto a Topic A and have a consumer (using @KafkaListener) subscribed to this topic ,

1. Does this spring listener would get all those 500 records and then is doing some kind of caching before passing one by one record to the method annotated with @KafkaListener or would it pull only one record at once and pass that to the method with @KafkaListener annotation

Answer 1

The @KafkaListener is based on the KafkaMessageListenerContainer and, in turn, is fully based on the ConsumerRecords<K, V> org.apache.kafka.clients.consumer.Consumer.poll(Duration timeout) API.

The option you mention has nothing to do with Spring for Apache Kafka. You would deal with the same behavior even without Spring.

See that returned ConsumerRecords for more info how records are fetched from Kafka.

With Kafka t really doesn't matter how we fetch records. Only an offset commit matters. But that's different story. You need to understand for yourself that Spring for Apache Kafka is just a wrapper around standard Kafka Client. It doesn't make an opinion how to poll records from topics.