Search code examples
spring-cloud-dataflow

How to manage SCDF stream definition and properties?


I wonder if there is a best practice to store a stream DSL and the apps and deployer properties file associated in a SCM like Git ?
What should a Git project for a SCDF Stream look like ?
And how can we manage versions of a stream ?


Solution

  • It's been a while but, for sharing, here is what we did. Thanks fiidim, your answer helps us.

    Our Git projects look like this :

    • apps.properties : used to register the apps
    • definition.txt : the DSL, used to create the stream
    • arguments.json : deployment properties, used to deploy the stream
    • YAML files : one for each app of the stream, with business properties

    We use Spring Cloud Config (with scdf profile) for business properties of the apps, that's why we separate deployment and business properties.
    We call the SCDF server with the REST API to register the apps, create and deploy the stream.

    With a ticktock example :
    apps.properties :

    source.time=docker:springcloudstream/time-source-kafka:3.2.1
    sink.log=docker:springcloudstream/log-sink-kafka:3.2.1
    

    definition.txt :
    time | log

    arguments.json :

    {  
      "version.time": "3.2.1",  
      "version.log": "3.2.1",  
      "app.*.spring.profiles.active": "scdf",  
      "app.*.spring.cloud.config.uri": "http://my-scc-server/config",  
      "deployer.*.kubernetes.imagePullPolicy=Always"  
    }
    

    mystream-log.yml
    log.expression: #root

    mystream-time.yml

    spring:
      cloud:
       stream:
         poller:
           time-unit: SECONDS
           fixed-delay: 10