Spring Kafka Consumers - Concurrent Listeners vs Non Concurrent Listeners in the Cloud

Question

I have created a Kafka consumer using Spring Kafka and deployed it in cloud foundry. The topic has 10 partitions. I am planning on scaling the app to 10 instances so that each instance can consume messages from one partition. Spring Kafka supports concurrent message listener container as well which I guess supports creation of more than 1 thread to consume from each partition. So for instance if I have 5 consumer instances, each consumer instance may have 2 threads consuming from partition each. Since I am anyway planning to create 1 app instance for each partition , is there any advantage of using concurrent consumers?

Since my priority is fast processing of the messages in the topic , I thought providing 1 app instance for each partition will give enough resources for the messages in each partition to be processed.

Answer 1

That's correct, you only need to have 10 single-threaded consumers in the same consumer group. Since you have exactly 10 partitions, the rebalance mechanism in Kafka will distribute them between your 10 instances.

The concurrency in that container will be unnecessary. Consider to use KafkaMessageListenerContainer instead.