In docker-compose, my init-kafka image is failing to list topics on my kafka image. How do I fix this? (With KRaft, not Zookeeper!)

(I know this question is similar to this other one, but the other one uses Zookeeper which is deprecated.)

I have this simple docker-compose:

services:
  kafka:
    container_name: dotnetbus-kafka
    image: apache/kafka:3.8.0
    ports:
      - 9092:9092

  init-kafka:
    container_name: dotnetbus-init-kafka
    image: apache/kafka:3.8.0
    depends_on:
      - kafka
    command: sleep 1000

NOTE: I'm looking for a solution that uses KRaft, not Zookeeper (because the latter is deprecated).

Eventually, I want to put a create-topic command in the init-kafka images instead of the sleep, but first I need to get those commands to actually work.

When I shell into the init-kafka image and try to list topics, this happens:

$ /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:9092 --list
[2024-09-10 03:17:03,924] WARN [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-09-10 03:17:04,027] WARN [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
...
(repeats infinitely)

I've tried a bunch of variations of that command. I must be understanding docker wrong, or Kafka wrong, or both.

However, when I run a similar command from my host machine (to localhost:9092) it works fine.

What am I missing? Why can't I get init-kafka to reach kafka?

Solution

With some help from the comments and this example, I've got it working.

Things I learned:

For KRaft, the compose environment needs to configure a "controller" listener, in addition to any other listeners.
Kafka will treat the config as Zookeeper unless it sees KAFKA_CONTROLLER_QUORUM_VOTERS.
These last 4 vars on my broker image are absolutely necessary. I honestly don't know what they do right now, but the default values didn't work (subscriptions always failed), and the error messages I was getting were 100% not-helpful for me with my present knowledge:
```
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
```

My working compose config

The following compose config configures 3 listeners:

a controller listener on docker-internal port 29092
a broker listener on docker-internal port 19092
a broker listener on host-exposed port 9092

The kafka1 service is the broker, and kafka-init creates topics on it.

services:
  kafka1:
    container_name: dotnetbus-kafka
    image: apache/kafka:3.8.0
    ports:
      - 9092:9092
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,INTERBROKER:PLAINTEXT,HOST:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'HOST://localhost:9092,INTERBROKER://kafka1:19092'
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:29092'
      KAFKA_LISTENERS: 'HOST://:9092,INTERBROKER://:19092,CONTROLLER://:29092'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'INTERBROKER'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

  kafka-init:
    container_name: dotnetbus-init-kafka
    image: apache/kafka:3.8.0
    depends_on:
      - kafka1
    command:
      - /bin/sh
      - -c
      - |
        sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --list
        echo -e 'Creating kafka topics'
        sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --create --if-not-exists --topic trade-request --replication-factor 1 --partitions 1
        sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --create --if-not-exists --topic trade-status --replication-factor 1 --partitions 1
        echo -e 'Successfully created the following topics:'
        sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --list

(Note: The depends_on ought to accompanied by a good health check, which I clearly haven't done in this config.)