Search code examples
spring-webfluxproject-reactorspring-cloud-streamspring-cloud-stream-binder-kafkareactor-kafka

How spring-cloud-stream Kafka Reactive stream partition assignment working with concurrent processing


For example, I configure a topic have 2 partitions, but in my application with 1 instance, I use Flux.parallel(10) to consume the message, and it has 1000 message lag on that topic, what will happen?

  1. Will it poll 10 messages per time? from 2 partitions or 1 partition?
  2. Only poll 2 message and 1 partition each?

I want to know how it works, so I could configure it right with the capability of large throughput and consume sequence

BTW I found this issue, but now answer there


Solution

  • It's better to use multiple receivers instead.

    Using parallel can cause problems with out of order offset commits.