Message ordering of ReactiveKafkaConsumerTemplate receiveAutoAck

Question

i am asking myself if the ReactiveKafkaConsumerTemplate of the spring-kafka project does guarantee the correct ordering of messages. I read the documentation of the reactor-kafka project and it states that messages should be consumed using the concatMap operator, but the ReactiveKafkaConsumerTemplate uses the flatMap operator at least in case of the receiveAutoAck method here:

https://github.com/spring-projects/spring-kafka/blob/master/spring-kafka/src/main/java/org/springframework/kafka/core/reactive/ReactiveKafkaConsumerTemplate.java#L69

Reference documentation of the reactor-kafka project: https://projectreactor.io/docs/kafka/release/reference/#_auto_acknowledgement_of_batches_of_records

I am interested in using receiveAutoAck as it seems to be the most simpelst and comfortable approach, which suffices my use case. The only way to overcome this behaviour of the receiveAutoAck method seems to subclass the ReactiveKafkaConsumerTemplate and overwrite this behaviour. Is this correct?

Answer 1

I don't think it really matters here because internally the source of data for us is Flux.fromIterable(consumerRecords) which cannot lose its order because of an iterator therefore how hard we wouldn't try to process them in parallel, we still would get the order in one iterator. Yes, the order in between iterators we flatten is really unpredictable, but this doesn't matter for us since we worry about an order withing a single partition, nothing more.

Nevertheless I think we definitely need to fix that for the mentioned concatMap() to avoid such a confusion in the future. Feel free to provide a contribution on the matter!