Search code examples
spring-cloud-streamspring-cloud-dataflow

spring-cloud-stream: Stream deployments fail on spring-cloud-dataflow-server-cf

In our use-case we need to create and deploy a Message Stream application to our private Cloud Foundry environment. The stream application is meant to listen on a RabbitMq queue and trigger a task, as defined in the stream.

To achieve the same, we using cloud-dataflow-server-cloudfoundry by running it locally on Mac.

Below are the set of commands used to achieve it:-

  • Set Maven remote repo export MAVEN_REMOTE_REPOSITORIES_REPO1_URL=https://ftl.abc.com/NEXUS

  • Setup Cloud Foundry destination -

    export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_URL=https://api.paas.abc-intl.com/
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_ORG=Paas-Apps
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_SPACE=DEV
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_DOMAIN=paas.abc-intl.com
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_USERNAME=abc@xyz.com
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_PASSWORD=password
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_SKIP_SSL_VALIDATION=true
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_SERVICES=rabbitmq_server
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_BUILDPACK=java_buildpack_offline
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_MEMORY=512
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_TASK_MEMORY=512
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_URL=https://api.paas.abc-intl.com/
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_ORG=Paas-Apps
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_SPACE=DEV
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_DOMAIN=paas.abc-intl.com
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_USERNAME=abc@xyz.com
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_PASSWORD=password
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_SKIP_SSL_VALIDATION=true
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_SERVICES=rabbitmq_server
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_BUILDPACK=java_buildpack_offline
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_MEMORY=512
export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_TASK_MEMORY=512

  • Start Dataflow Server -

    1. Build Dataflow Server CF with spring cloud properties as:

      • spring.name=dataflow
      • spring.cloud.config.uri=http://localhost:8888
      • spring.cloud.dataflow.features.tasksEnabled=true
      • spring.cloud.dataflow.features.streamsEnabled=true
      • spring.cloud.deployer.cloudfoundry.task.taskTimeout=360

    2. Start DF Server CF locally - java -Djavax.net.ssl.trustStore=/Users/apache-maven-3.3.9/conf/keystore.jks -Djavax.net.ssl.trustStorePassword=password -jar spring-cloud-dataflow-server-cloudfoundry-1.1.0.BUILD-SNAPSHOT.jar

  • Start DF Shell
  • Import starter apps - stream-applications-rabbit-maven
  • Register apps -
    1. app register --name task-processor --type processor --uri maven://org.springframework.cloud.stream.app:tasklaunchrequest-transform-processor-rabbit:jar:1.1.0.BUILD-SNAPSHOT
    2. app register --name task-launcher-local --type sink --uri maven://org.springframework.cloud.stream.app:task-launcher-local-sink-rabbit:jar:1.0.4.RELEASE
  • Create & Deploy Stream - stream create myMessageStream --definition "rabbit --rabbit.queues=cloud-stream-source --rabbit.requeue=true --spring.rabbitmq.host=host.abc.com --spring.rabbitmq.port=5672 --spring.cloud.stream.bindings.output.contentType='text/plain' | task-processor --uri=maven://com.example:thumbnail-generator:0.0.1-SNAPSHOT | task-launcher-local" --deploy

However the deployment fails with TimeoutException but with no clear reason either in the Dataflow server logs or the applications' logs.

The created application in cloud foundry space are all showing Stopped on the CF console.

This left me clueless and I've got couple of questions on this to understand the possible root cause:

  1. In case of a successful Stream deployment, what shall be the status of applications deployed to CF by Stream? Do these show up as started or stopped?
  2. The name of the deployed applications has couple of random words embedded in it e.g. dataflow-nonvaluable-overapprehension-myMessageStream-rabbit. In my understanding that is the name given to the Dataflow Server instance and is inserted to make a reference. Is there a way to customise this name? That will enable me to know the name of applications in advance and hen run a tail on their logs.

Any thoughts/help will be appreciated.

Solution

  • In case of a successful Stream deployment, what shall be the status of applications deployed to CF by Stream? Do these show up as started or stopped?

    The stream list command will display the status. If the stream deployment is successful, the status should be deployed.

    The name of the deployed applications has couple of random words embedded in it e.g. dataflow-nonvaluable-overapprehension-myMessageStream-rabbit. In my understanding that is the name given to the Dataflow Server instance and is inserted to make a reference. Is there a way to customise this name? That will enable me to know the name of applications in advance and hen run a tail on their logs.

    You can override the random name generation in the app name by disabling this property by doing: export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_ENABLE_RANDOM_APP_NAME_PREFIX=false

    You can control the app name using the deployer property which sets a specific prefix:

    export SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_APP_NAME_PREFIX=myprefix

    You can try increasing the deployer properties SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_STAGING_TIMEOUT and SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_STARTUP_TIMEOUT to see if the app requires some more time to stage/startup.