Can I use KafkaItemReader in Spring Batch to read from multiple topics?

Question

It's easy to use KafkaItemReader to read messages from one topic:

return new KafkaItemReaderBuilder<String, Activity>()
    .consumerProperties(toProperties(of(
            ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress,
            ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class,
            ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class
    )))
    .pollTimeout(ofMillis(600))
    .topic(checkOutTopicName)
    .build();

However, topic method only permits to assign a single topic. If I want to read checkInTopicName as well, I have create another bean for check in as well. The behavior of the job for check in is not that different from the one for check out and I think they should be more coherent.

Is there a way to read multiple topics with KafkaItemReader?

---

I feel that it might be an XY problem actually. In the next iteration I am going to use partitions check-in-partition and check-out-partition on a single topic instead of two separate topics. I think it will make my job more coherent, but for now I want to know how it could be done with two topics instead of partitions.

Answer 1

It is not possible, since the Batch reader uses the assignment API of Kafka, therefore accepts a static list of TopicPartition instances, which get created with that single topic name.

The solution would be to use spring-kafka, not spring-batch , where you can read from a list of topics, including with a regex pattern.