Search code examples
apache-kafkakafka-consumer-apiapache-kafka-streams

Can we control number of consumers in a consumer group through code in Kafka?


Can we control the number of consumers in a consumer group through code in Kstream application?

In my application, I have 8 instances with 20 threads each, how many consumers will be created in my consumer group?

Kafka topic partitions = 480

Will it be 8 instances * 20 threads = 160 consumers?


Solution

  • You control the number of threads in one application

    And you control the number of instances by running the application process in parallel (JAR, Docker, etc).

    Yes, you multiply them together to get total consumers per application.id, although, Kafka Streams can be consuming multiple topics at once, not all with the same partition count