Kafka 3.4 with Kraft without Zookeeper with 3 brokers

Question

I need 1 cluster kafka with 3 borkers with kraft. I try with Below docker config:

Config for 1st broker:

version: "3"
services:
 kafka-1:
  image: "bitnami/kafka:3.4.0"
  hostname: kafka-1
  environment:
   - ALLOW_ANONYMOUS_LOGIN=yes
   - KAFKA_ENABLE_KRAFT=yes
   - KAFKA_CFG_PROCESS_ROLES=broker,controller
   - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
   - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=kraft:PLAINTEXT,CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT
   - KAFKA_BROKER_ID=1
   - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9094
   - ALLOW_PLAINTEXT_LISTENER=yes
   - BITNAMI_DEBUG=yes
   - KAFKA_KRAFT_CLUSTER_ID=OTMwNzFhYTY1ODNiNGE5OT
   - KAFKA_CFG_KRAFT_REPLICATION_FACTOR=3
   - KAFKA_CFG_ADVERTISED_LISTENERS=kraft://:9093,INTERNAL://kafka-1:9092
   - KAFKA_CFG_LISTENERS=kraft://:9093,CONTROLLER://kafka-1:9094,INTERNAL://:9092
   - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL
   - KAFKA_CFG_DEFAULT_REPLICATION_FACTOR=3
  ports:
   - "9101:9092"
   - "9102:9093"
   - "9103:9094"

Config for 2nd broker:

 kafka-2:
   image: "bitnami/kafka:3.4.0"
   hostname: kafka-2
   environment:
     - ALLOW_ANONYMOUS_LOGIN=yes
     - KAFKA_ENABLE_KRAFT=yes
     - KAFKA_CFG_PROCESS_ROLES=broker,controller
     - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
     - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=kraft:PLAINTEXT,CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT
     - KAFKA_BROKER_ID=2
     - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=2@kafka-2:9094
     - ALLOW_PLAINTEXT_LISTENER=yes
     - BITNAMI_DEBUG=yes
     - KAFKA_KRAFT_CLUSTER_ID=OTMwNzFhYTY1ODNiNGE5OT
     - KAFKA_CFG_KRAFT_REPLICATION_FACTOR=3
     - KAFKA_CFG_ADVERTISED_LISTENERS=kraft://:9093,INTERNAL://kafka-2:9092
     - KAFKA_CFG_LISTENERS=kraft://:9093,CONTROLLER://kafka-2:9094,INTERNAL://:9092
     - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL
     - KAFKA_CFG_DEFAULT_REPLICATION_FACTOR=3
   ports:
     - "9095:9092"
     - "9096:9093"
     - "9097:9094"

Config for 3rd broker:

 kafka-3:
   image: "bitnami/kafka:3.4.0"
   hostname: kafka-3
   environment:
     - ALLOW_ANONYMOUS_LOGIN=yes
     - KAFKA_ENABLE_KRAFT=yes
     - KAFKA_CFG_PROCESS_ROLES=broker,controller
     - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
     - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=kraft:PLAINTEXT,CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT
     - KAFKA_BROKER_ID=3
     - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=3@kafka-3:9094
     - ALLOW_PLAINTEXT_LISTENER=yes
     - BITNAMI_DEBUG=yes
     - KAFKA_KRAFT_CLUSTER_ID=OTMwNzFhYTY1ODNiNGE5OT
     - KAFKA_CFG_KRAFT_REPLICATION_FACTOR=3
     - KAFKA_CFG_ADVERTISED_LISTENERS=kraft://:9093,INTERNAL://kafka-3:9092
     - KAFKA_CFG_LISTENERS=kraft://:9093,CONTROLLER://kafka-3:9094,INTERNAL://:9092
     - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL
     - KAFKA_CFG_DEFAULT_REPLICATION_FACTOR=3

   ports:
     - "9098:9092"
     - "9099:9093"
     - "9100:9094"

All config are in same file.

When i run this docker compose i see 3 clusters not 1 cluster with 3 brockers.How can i configure to make 1 cluster kafka? CORRECTION OF MY DOCKER COMPOSE TO GET 1 CLUSTER

Answer 1

Tbh I don't know this bitnami/kafka:3.4.0 Docker image you are using but I guess that it has some scripts to get the env vars you are listing and it will do some work for you to configure the cluster (i.e. formatting the storage starting from the KAFKA_KRAFT_CLUSTER_ID env var). Say that, what I guess is an error is the KAFKA_CFG_CONTROLLER_QUORUM_VOTERS. You are listing just one broker for each configuration while it should list all the 3 brokers which could be part of the controllers quorum. So for all of them you should have something like: KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9094,2@kafka-2:9094,3@kafka-3:9094. Of course, the three brokers should be able to communicate across the newtwork to have it working (so the various kafka-X should be discoverable and adressable via DNS. I don't use Docker Compose, maybe it works out of box).