Search code examples
spring-bootapache-kafkaapache-kafka-streamsspring-cloud-streamspring-kafka

Spring and Kafka: Using the same binding for Kafka Producer and Kafka Streams


Is this possible to use the same binding for producing messages via Kafka Producer and for sending results of stream processing by Kafka Streams?

Right now I have them separated as follows:

    @Output(TOPIC_X)
    MessageChannel xOutput();

    @Output(TOPIC_X_ONE)
    KStream<?,?> xOneOutput();

When the first one is used by producer:

channels
.xOutput()
.send(MessageBuilder
      .withPayload(x)
      .setHeader(KafkaHeaders.MESSAGE_KEY, x.getId()).build());

And second one by kafka streams:

@StreamListener
@SendTo(TOPIC_X_ONE)
public KStream<?,?> process(@Input(INPUT_TOPIC) KStream<String,String> inputStream){
    //ommited for clarity
}

And in application.yml I have:

spring.cloud.stream.bindings:
  topic-x:
    destination: mytopic
    producer:
      useNativeEncoding: true
  topic-x-one:
    destination: mytopic
    producer:
      useNativeEncoding: true

Is it really necessary to have them separated if they both have the same destination and both are outputs?


Solution

  • Yes, it is necessary; the bindings are handled by two different binder implementations - and a different kafka client.