I have been using apache kafka-clients(2.3.1 version to be precise) libraries for creating kafka consumers where the one partition - one consumer thread was achieved by following calculation:
no of consumer thread on a compute * no of computes = number of partitions for the topic
It used to be manual scale and hence when the number of computes needed to be crease, no of consumer threads running on one compute was accordingly decreased.
How do we achieve this using org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory.
I am trying to use spring kafka 2.5.8 release. The application is running on k8s cluster with auto scaling capability. Let's say if I have max and min pods set to 4, so ideally
4 X number of consumer threads = number of partitions for the topic
How is this number of consumer threads configured . Is it through this:
org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory.setConcurrency() . Can someone please guide.
Yes; or the concurrency property on @KafkaListener which overrides the factory's concurrency.
If you change it at runtime, it won't take effect unless you stop() and start() the container.