How to make a new Kafka Consumer to read offset from latest

Question

I am using sping-kafka to read from a topic that has thousands of events and have multiple consumer groups on the same topic.

auto.offset.reset is set to "earliest" as Kafka consumer configurations

config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

I want to add a new consumer group that should skip existing events from the topic.

Is there any way to override auto.offset.reset to latest at Kafka handler level without impacting other consumers?

Answer 1

See the documentation:

https://docs.spring.io/spring-kafka/docs/current/reference/html/#annotation-properties

Starting with version 2.2.4, you can specify Kafka consumer properties directly on the annotation, these will override any properties with the same name configured in the consumer factory. You cannot specify the group.id and client.id properties this way; they will be ignored; use the groupId and clientIdPrefix annotation properties for those.

The properties are specified as individual strings with the normal Java Properties file format: foo:bar, foo=bar, or foo bar.

@KafkaListener(topics = "myTopic", groupId = "group", properties = {
    "max.poll.interval.ms:60000",
    ConsumerConfig.MAX_POLL_RECORDS_CONFIG + "=100"
})

If you are not using annotations, you can use a ContainerCustomizer to set the properties on a particular container.