Search code examples
spring-bootapache-kafkakafka-consumer-apispring-kafka

Kafka multiple consumer from different partitions

I have 4 partitions and 4 consumers(A,B,C,D for example). How to configure which consumer will read from which partition using consumer groups. I am using Kafka with Spring boot.

Solution

  • By default, kafka will automatically assign the partitions; if you have 4 consumers in the same group, they will eventually get one partition each. There are properties to configure kafka so it won't immediately do the allocation while you bring up your consumers.

    You can also assign the partitions yourself.

    Using

    public ContainerProperties(TopicPartitionInitialOffset... topicPartitions)

    if you are building the container yourself, or

    @KafkaListener(id = "baz", topicPartitions = @TopicPartition(topic = "${topic}",
            partitions = "${partition}"))

    if you are using @KafkaListener.