KStreams with Spring Cloud Stream (Kafka Binder) - Setting Partitions

Question

I wrote a small KStreams Processor using Spring Cloud Stream - https://github.com/sandeep540/kafka-streams-spring3-cloud-java17

Here I am reading from Kafka Topic "input-topic" with 5 Partitions, processing it and sending it to another topic

When I run the program , I only see 1 thread for all 5 partitions, inspite of providing partition properties in application.yml

My requirement is to have 5 threads running (1 for each Partition), what is that I am missing, any help here is appreciated

Answer 1

You can configure the number of stream threads using the num.stream.threads property:

This specifies the number of stream threads in an instance of the Kafka Streams application. The stream processing code runs in these threads. For more info about the Kafka Streams threading model, see Threading Model.

https://docs.confluent.io/platform/current/streams/developer-guide/config-streams.html#num-stream-threads

For more information, see Confluent's excellent documentation on the Kafka Streams threading model.