I'm trying to complete the first spring cloud dataflow tutorial and I'm not getting the result in the tutorial.
https://dataflow.spring.io/docs/stream-developer-guides/streams/
The tutorial has me use curl to a http source and see the result in the log sink with a tail of a file of stdout.
I do not see the result. I see the startup in the log.
I tail the log docker exec -it skipper tail -f /path/from/stdout/textbox/in/dashboard
I enter curl http://localhost:20100 -H "Content-type: text/plain" -d "Happy streaming"
all I see is
2020-10-05 16:30:03.315 INFO 110 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka version : 2.0.1
2020-10-05 16:30:03.316 INFO 110 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka commitId : fa14705e51bd2ce5
2020-10-05 16:30:03.322 INFO 110 --- [ main] o.s.s.c.ThreadPoolTaskScheduler : Initializing ExecutorService
2020-10-05 16:30:03.338 INFO 110 --- [ main] s.i.k.i.KafkaMessageDrivenChannelAdapter : started org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter@106faf11
2020-10-05 16:30:03.364 INFO 110 --- [container-0-C-1] org.apache.kafka.clients.Metadata : Cluster ID: 2J0QTxzQQmm2bLxFKgRwmA
2020-10-05 16:30:03.574 INFO 110 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 20041 (http) with context path ''
2020-10-05 16:30:03.584 INFO 110 --- [ main] o.s.c.s.a.l.s.k.LogSinkKafkaApplication : Started LogSinkKafkaApplication in 38.086 seconds (JVM running for 40.251)
2020-10-05 16:30:05.852 INFO 110 --- [container-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-3, groupId=http-ingest] Discovered group coordinator kafka-broker:9092 (id: 2147482646 rack: null)
2020-10-05 16:30:05.857 INFO 110 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-3, groupId=http-ingest] Revoking previously assigned partitions []
2020-10-05 16:30:05.858 INFO 110 --- [container-0-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1 : partitions revoked: []
2020-10-05 16:30:05.858 INFO 110 --- [container-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-3, groupId=http-ingest] (Re-)joining group
2020-10-05 16:30:08.943 INFO 110 --- [container-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-3, groupId=http-ingest] Successfully joined group with generation 1
2020-10-05 16:30:08.945 INFO 110 --- [container-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-3, groupId=http-ingest] Setting newly assigned partitions [http-ingest.http-0]
2020-10-05 16:30:08.964 INFO 110 --- [container-0-C-1] o.a.k.c.consumer.internals.Fetcher : [Consumer clientId=consumer-3, groupId=http-ingest] Resetting offset for partition http-ingest.http-0 to offset 0.
2020-10-05 16:30:08.981 INFO 110 --- [container-0-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1 : partitions assigned: [http-ingest.http-0]
No Happy streaming
Any suggestions?
Thank you for trying out the developer guides!
From what I can tell, it appears the http | log stream definition in SCDF is submitted without an explicit port. When that is the case, a port gets randomly assigned by Spring Boot when the http-source and log-sink applications start.
If you navigate to your http-source application logs, you will see the application port listed, and that is the port you'd use on the CURL command.
There's this following note about this in the guide for your reference.
If you use the local Data Flow Server, add the following deployment property to set the port to avoid a port collision.
Alternatively, you can deploy the stream with an explicit port in the definition. For example: http --server.port=9004 | log. With that, your CURL would then be:
curl http://localhost:9004 -H "Content-type: text/plain" -d "Happy streaming"