Search code examples
apache-kafkaspring-kafka

Question about using the @KafkaListener autoStartup = "false" for multiple topics


With the following

@KafkaListener(id = "id1", autoStartup = "false", topics = { "topic1", "topic2" } spring.kafka.consumer.auto-offset-reset=earliest

Assume there are multiple messages in each topic before .start is run

When

KafkaListenerEndpointRegistry.getListenerContainer("id1").start();

is called is there a guarantee that all messages from topic1 will get processed before topic2 (that is what I am seeing), and what happens if messages are sent to topic1 while topic2 is getting processed.

+++++++++++++++++++++++++++++++++++++++++++++++++

Edit

Ran the following test, each topic has a single partition. Before the test was run there were 10 messages in topic1 and 10 messages in topic2. Ran the code and let the 10 topic1 messages get processed but while the topic2 messages where getting processed I sent in more messages to topic1 but there where not processed by the listener until all the preexisting messages from topic2 had been processed.

So it seems like the messages are processed in order from the topics property array and any new messages are not processed until the existing ones are processed.

The order the messages where processed in topic1 message 1 topic1 message 2 ... topic1 message 10 topic2 message 1 topic2 message 2 ... ... sent message 11 to topic 1 at this time ... topic2 message 10 topic1 message 11


Solution

  • There is no such guarantee.

    See answers to this question.

    Kafka gives you only the guarantee of messages ordering inside a partition. ...

    Regarding your use case with two topics there is no relation between subscription order to the topics and messages ordering even because ...