Spring Cloud Dataflow with multiple Kafka binders

Question

I'm trying to use Spring Cloud Dataflow to bridge two Kafka clusters (essentially a fancy MirrorMaker instance) using the Bridge app. As covered in the docs, I've defined two binders. Kafka-qa1 should be the default, and kafka-qa2 can be provided in the definition or deployment properties as the output binder e.g.: app.bridge.spring.cloud.stream.bindings.output.binder=kafka-qa2

My SCDF application.yaml contains both binders:

spring:
  cloud:
    dataflow:
      applicationProperties:
        stream:
          spring:
            cloud:
              stream:
                defaultBinder: kafka-qa1
                binders:
                  kafka-qa1:
                    type: kafka
                    environment:
                      spring:
                        brokers: qa-1.example.com:9093
                        zk-nodes: qa-1.example.com:2181
                  kafka-qa2:
                    type: kafka
                    environment:
                      spring:
                        brokers: qa-2.example.com:9093
                        zk-nodes: qa-2.example.com:2181

However it seems to be ignoring the output binder. I've also kept the section for use with a single binder in my config (below). If I remove it the defaultBinder option doesn't seem to work and it reverts to localhost.

kafka:
 binder:
  brokers: qa-1.example.com:9093

Any ideas or examples to point me to for connecting multiple Kafka clusters with the Bridge app?

Answer 1

It looks like the environment token is missing spring.cloud.stream.kafka.binder prefix for brokers and zk-nodes. Please see below.

spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.binders.kafka-qa1.type=kafka

spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.binders.kafka-qa1.environment.spring.cloud.stream.kafka.binder.brokers=qa-1.example.com:9093

spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.binders.kafka-qa1.environment.spring.cloud.stream.kafka.binder.zkNodes=qa-1.example.com:2181

spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.binders.kafka2.type=kafka

spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.binders.kafka-qa2.environment.spring.cloud.stream.kafka.binder.brokers=qa-2.example.com:9093

spring.cloud.dataflow.applicationProperties.stream.spring.cloud.stream.binders.kafka-qa2.environment.spring.cloud.stream.kafka.binder.zkNodes=qa-2.example.com:2181