I'm new to kafka. How can I define two(or more) Kafka consumers using configuration .properties or .yml? I'm interested in using the spring.kafka.* application's properties and also I would like to specify two different properties for two consumers using this configuration files. For example, consumerA will have spring.kafka.bootstrap-servers=localhost:9090 and consumerB spring.kafka.bootstrap-servers=localhost:9091. I have seen examples online using multiple @KafkaLister with a single application.yml where all the common properties of the @KafkaLister (consumer) beans are defined in application.yml. This is ok concept but in case there are more consumers and they have completely different configurations then all of the configs needs to be put in the @KafkaListener annotation and this will make the class long and hard to read and the .yml obsolete. Instead, I would like something like this:
spring:
kafka:
consumer1:
auto-offset-reset: earliest
spring.kafka.bootstrap-servers=localhost:9091
kafka:
consumer2:
spring.kafka.bootstrap-servers=localhost:9092
auto-offset-reset: latest
kafka:
consumer3:
spring.kafka.bootstrap-servers=localhost:9093
auto-offset-reset: latest
And also how would I then connect this configuration to the appropriate beans? I could surely define the consumer beans and then use my custom configuration files to create as many different consumers as I would like but it seems to me that I would be reinventing the wheel.
Thanks
No; you can't declare consumers like that, you have to use @KafkaListener annotations on methods or create listener containers/listeners yourself.
You can override the bootstrap servers in the listener itself...
@KafkaListener(... properties="bootstrap.servers:${some.property}")
void listen(...) {
}
You can programmatically make multiple consumers from user properties, using the technique described here...