Search code examples
spring-xdspring-cloud-streamspring-cloud-dataflow

Spring Data Flow w/ 2 sources feeding one processor/sink

I'm looking for some advice on setting up a Spring Data Flow stream for a specific use case.

My use case:

I have 2 RDBMS and I need to compare the results of queries run against each. The queries should be run roughly simultaneously. Based on the result of the comparison, I should be able to send an email through a custom email sink app which I have created.

I envision the stream diagram to look something like this (sorry for the paint): enter image description here

The problem is that SDF does not, to my knowledge, allow a stream to be composed with 2 sources. It seems to me that something like this ought to be possible without pushing the limits of the framework too far. I'm looking for answers that provide a good approach to this scenario while working within the SDF framework.

I am using Kafka as a message broker and the data flow server is using mysql to persist stream information.

I have considered creating a custom Source app which polls two datasources and sends the messages on the output channel. This would eliminate my requirement of 2 sources, but it looks like it would require a significant amount of customization of the jdbc source application.

Thanks in advance.

Solution

  • I have not really tried this, but you should be able to use named destinations to achieve that. Take a look here: http://docs.spring.io/spring-cloud-dataflow/docs/current-SNAPSHOT/reference/htmlsingle/#spring-cloud-dataflow-stream-advanced

    stream create --name jdbc1 --definition "jdbc > :dbSource"

    stream create --name jdbc2 --definition "jdbc > :dbSource"

    stream create --name processor --definition ":dbSource > aggregator | sink"