Search code examples
kubernetesapache-kafkakafka-consumer-apibitnami

kafka configuration file appears to be for a legacy cluster

I'm getting an error when deploying a simple kafka using kafka 3.3

kafka 12:58:07.67 WARN ==> KAFKA_KRAFT_CLUSTER_ID not set - If using multiple nodes then you must use the same Cluster ID for each one
kafka 12:58:13.51 INFO ==> Generated Kafka cluster ID 'xxxx'
kafka 12:58:13.51 INFO ==> Formatting storage directories to add metadata...
The kafka configuration file appears to be for a legacy cluster. Formatting is only supported for clusters in KRaft mode.

This is the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: kafka-deployment
  name: kafka
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
        - env:
            - name: ALLOW_PLAINTEXT_LISTENER
              value: "yes"
            - name: KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE
              value: "true"
            - name: KAFKA_CFG_NODE_ID
              value: "1"
            - name: KAFKA_CFG_BROKER_ID
              value: "1"
            - name: KAFKA_CFG_CONTROLLER_QUORUM_VOTERS
              value: xxx
            - name: KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP
              value: xxx
            - name: KAFKA_CFG_CONTROLLER_LISTENER_NAMES
              value: xxx
            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: xxx
            - name: KAFKA_CFG_PROCESS_ROLES
              value: xxx
            - name: KAFKA_CFG_LISTENERS
              value: xxx
            - name: KAFKA_CFG_ADVERTISED_LISTENERS
              value: xxx
          image: bitnami/kafka:3.3

I see that the image in bitnami/kafka for kafka:3.3 changed few daysa go but I'm not sure if the issue is related or if missing any requiered env var. It was working fine few days ago.

I was getting similar warnings but kafka always started successfully.

Solution

  • You are using bitnami/kafka:3.3 which it might point to a different minor relase in bitnami.

    What if you depend on a specific revision of an image? For these scenarios, Bitnami also attaches a static (immutable) tag to each revision. In this example, the 3.4.6-r24 tag refers to Redmine 3.4.6 revision 24 and using this tag ensures that users always get exactly the same image every time.

    In your case bitnami/kafka:3.3 image changed few hours ago and points to the commit 6fa209c22f77 which is the same as bitnami/kafka:3.3.2.

    If you want to make sure that you are always going to use the same kafka version from bitnami, just use immutable tags.

    In your specific case you can try using bitnami/kafka:3.3.1-debian-11-r38 as immutable tag.