Healthcheck not working at all when using docker-compose (My service do not wait for Kafka to be started before launching)

Question

I have three services on my docker-compose:

version: '3.4'
  setup-topics:
    image: 'bitnami/kafka:2'
    hostname: setup-topics
    container_name: setup-topics
    command: "bash -c 'echo Waiting for Kafka to be ready... && \
                       ./opt/bitnami/kafka/bin/kafka-topics.sh --create --if-not-exists --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic orders && \
                       ./opt/bitnami/kafka/bin/kafka-topics.sh --create --if-not-exists --zookeeper zookeeper:2181 --partitions 1 --replication-factor 1 --topic redis'"
    environment:
      KAFKA_BROKER_ID: ignored
      KAFKA_ZOOKEEPER_CONNECT: ignored
    depends_on:
      - kafka

  kafka:
    container_name: kafka
    hostname: kafka
    image: 'bitnami/kafka:2'
    ports:
      - '9092:9092'
      - '29092:29092'
    volumes:
      - 'kafka_data:/opt/kafka'
      - './Ping.jar:/Ping.jar'
    environment:
      - KAFKA_HEAP_OPTS=-Xms1g -Xmx1g
      - KAFKA_JVM_PERFORMANCE_OPTS=-Xms512m -Xmx512M
      - KAFKA_BROKER_ID:1
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,PLAINTEXT_HOST://:29092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-server:9092,PLAINTEXT_HOST://localhost:29092
    depends_on:
      - zookeeper

    healthcheck:
      test: ["CMD", "java", "-jar", "/Ping.jar", "localhost", "9092"]
      interval: 30s
      timeout: 10s
      retries: 4

  zookeeper:
    container_name: zookeeper
    hostname: zookeeper
    image: 'bitnami/zookeeper:3'
    ports:
      - '2181:2181'
    volumes:
      - 'zookeeper_data:/bitnami'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOOKEEPER_CLIENT_PORT=32181
      - ZOOKEEPER_TICK_TIME=2000

And here the Ping.java file (Found it on here on stackoverflow answer: Docker-Compose: How to healthcheck OpenJDK:8 container?):

import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Socket;

public class Ping {
    public static void main(String[] args) {
        if (args.length != 2) {
            System.exit(-1);
        }

        String host = args[0];
        int port = 0;

        try {
            port = Integer.parseInt(args[1]);
        } catch (NumberFormatException e) {
            e.printStackTrace();
            System.exit(-2);
        }

        try (Socket socket = new Socket()) {
            socket.connect(new InetSocketAddress(host, port), 10 * 1000);
            System.exit(0);
        } catch (IOException e) {
            System.exit(1);
        }
    }
}

Even with depends_on on the SETUP-TOPICS service to be dependent on Kafka in order to works, but he don't wait until Kafka is started before running and install new topics.

I can avoid this steps by using:

KAFKA_AUTO_CREATE_TOPICS_ENABLE=true

But for my developement purpose, I need to make it FALSE and create them one by one.

In top of this, I already tested with this command in the Healthcheck without requiring third party file:

healthcheck:
  test: ["CMD", "bash", "-c", "unset" , "JMX_PORT" ,";" ,"/opt/bitnami/kafka/bin/kafka-topics.sh","--zookeeper","zookeeper:2181","--list"]
  interval: 30s
  timeout: 10s
  retries: 4

And finally, here is the error message I am getting for both tries:

Waiting for Kafka to be ready...
Error while executing topic command : Replication factor: 1 larger than available brokers: 0.
[2020-06-01 15:06:28,809] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
 (kafka.admin.TopicCommand$)

I am aware that we can do it also using SLEEP command, but its not professional and if there will be performances issue on the server and Kafka take longer to start, this one will be missed and recieve again the same error as above.

I hear also about kafkacat (Which I didn't found yet an example on how to integrate it with docker-compose for this purpose).

I want to stay basic and use limited third party tools to achieve this goal, this is way I choosen JAVA file since the image already have Java dependency installed.

Hpe you understand my view, thank you in advance for your help.

Answer 1

Not clear why you need a JAR file. This should work just as well

test: ["CMD", "nc", "-vz", "localhost", "9092"]

The problem is simply waiting for kafka has not been waiting long enough before you run your commands. depends_on does not wait for healthcheck, AFAIK