Search code examples
javaspring-cloud-dataflow

Streams deploying stuck and then fails with no errors in the logs

I'm trying to deploy the following streams:

STREAM_2=:messages > filter --expression="#jsonPath(payload, '$.id')==1" | rabbit --queues=id_1 --host=rabbitmq --routing-key=id_1 --exchange=ex_1 --own-connection=true
STREAM_3=:messages > filter --expression="#jsonPath(payload, '$.id')==2" | rabbit --queues=id_2 --host=rabbitmq --routing-key=id_2 --exchange=ex_1
STREAM_4=:messages > filter --expression="#jsonPath(payload, '$.id')==3" | rabbit --queues=id_3 --host=rabbitmq --routing-key=id_3 --exchange=ex_1
STREAM_1=rabbit --queues=hello_queue --host=rabbitmq > :messages

Visualization:

enter image description here

I'm listening for a queue and then sending the message to a different queue depending on one of the message's attributes.

I'm running a local system, using this docker-compose.yml, but I switched to RabbitMQ instead of Kafka for communication.

When I deploy the streams, it takes a couple of minutes until the dataflow-server container reaches the max memory usage, and finally fails on random streams (and sometimes kills the container).

The logs (both stdout and stderr) don't show errors.

I'm running with the latest versions as follows:

DATAFLOW_VERSION=2.0.1.RELEASE SKIPPER_VERSION=2.0.0.RELEASE docker-compose up

Another thing I noticed, in the logs I keep getting:

2019-03-27 09:35:00.485 WARN 70 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 could not be established. Broker may not be available.

although I have nothing related to Kafka in my docker-compose.yml. Any ideas where it's coming from?

Relevant parts from my YAML:

version: '3'

services:
  mysql:
    image: mysql:5.7.25
    environment:
      MYSQL_DATABASE: dataflow
      MYSQL_USER: root
      MYSQL_ROOT_PASSWORD: rootpw
    expose:
      - 3306
  dataflow-server:
    image: springcloud/spring-cloud-dataflow-server:${DATAFLOW_VERSION:?DATAFLOW_VERSION is not set!}
    container_name: dataflow-server
    ports:
      - "9393:9393"
    environment:
      - spring.datasource.url=jdbc:mysql://mysql:3306/dataflow
      - spring.datasource.username=root
      - spring.datasource.password=rootpw
      - spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
      - spring.cloud.skipper.client.serverUri=http://skipper-server:7577/api
      - spring.cloud.dataflow.applicationProperties.stream.spring.rabbitmq.host=rabbitmq
    depends_on:
      - rabbitmq

  rabbitmq:
    image: "rabbitmq:3-management"
    ports:
      - "5672:5672"
      - "15672:15672"
    expose:
      - "5672"

  app-import:
    ...

  skipper-server:
    image: springcloud/spring-cloud-skipper-server:${SKIPPER_VERSION:?SKIPPER_VERSION is not set!}
    container_name: skipper
    ports:
    - "7577:7577"
    - "9000-9010:9000-9010"

volumes:
  scdf-targets:
Solution

  • Looks like I were a victim of the OOM killer. The container was crashing with an exit code of 137.

    The easiest solution for me now is giving Docker more memory:

    CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
9a0e3ff0beb8        dataflow-server     0.18%               1.293GiB / 11.71GiB   11.04%              573kB / 183kB       92.1MB / 279kB      49
2a448b3583a3        scdf_kafka_1        7.00%               291.6MiB / 11.71GiB   2.43%               4.65MB / 3.64MB     40.4MB / 36.9kB     73
eb9a70ce2a0e        scdf_rabbitmq_1     2.15%               94.21MiB / 11.71GiB   0.79%               172kB / 92.5kB      41.7MB / 139kB      128
06dd2d6a1501        scdf_zookeeper_1    0.16%               81.72MiB / 11.71GiB   0.68%               77.8kB / 99.2kB     36.7MB / 45.1kB     25
1f1b782ad66d        skipper             8.64%               6.55GiB / 11.71GiB    55.93%              3.63MB / 4.73MB     213MB / 0B          324

    The skipper container is now using 6.55GiB memory, if someone knows what it could be, I would be grateful.

    For now, I'm accepting my answer since it does provide a workaround, although I feel there could be a better solution than increasing the memory limit for Docker.

    EDIT:

    Looks like this is indeed the solution, from this GitHub issue:

    Stream components (parts of the pipe) are deployed as applications. Those applications are deployed into the Skipper container (as well as the Skipper application itself) since skipper deploys streams. The more applications that get deployed (parts of the pipe, streams, etc) the more memory is used.