Strimzi Kafka brokers not be created because of "ingress does not contain a valid IngressClass"

Question

I have a local k3s Kubernetes cluster (Traefik disabled) created by Rancher Desktop.

I am trying to set up Kafka using Ingress way based on this tutorial to make it accessible for clients running outside of Kubernetes.

In able to make it work, Ingress need enable SSL passthrough. I installed ingress-nginx by

helm upgrade \
  ingress-nginx \
  ingress-nginx \
  --install \
  --repo=https://kubernetes.github.io/ingress-nginx \
  --namespace=ingress-nginx \
  --create-namespace \
  --values=my-values.yaml

my-values.yaml

controller:
  extraArgs:
    enable-ssl-passthrough: true

I got my cluster IP 192.168.1.149 by:

➜ kubectl get node lima-rancher-desktop -o wide
NAME                   STATUS   ROLES                  AGE    VERSION        INTERNAL-IP    EXTERNAL-IP     OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
lima-rancher-desktop   Ready    control-plane,master   108m   v1.26.3+k3s1   192.168.5.15   192.168.1.149   Alpine Linux v3.16   5.15.96-0-virt   containerd://1.6.8

Then I deploy my Kafka by:

kubectl create namespace hm-kafka
kubectl apply --filename="https://strimzi.io/install/latest?namespace=hm-kafka" --namespace=hm-kafka
kubectl apply --filename=my-kafka-persistent.yaml --namespace=hm-kafka

my-kafka-persistent.yaml (based on kafka-persistent.yaml):

---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: hm-kafka
spec:
  kafka:
    version: 3.4.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: ingress
        tls: true
        configuration:
          bootstrap:
            host: kafka-bootstrap.192.168.1.149.nip.io
          brokers:
          - broker: 0
            host: kafka-broker-0.192.168.1.149.nip.io
          - broker: 1
            host: kafka-broker-1.192.168.1.149.nip.io
          - broker: 2
            host: kafka-broker-2.192.168.1.149.nip.io
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.4"
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 100Gi
          deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

However, this time Kafka broker are not created.

When I check ingress-nginx log, it shows:

➜ kubectl logs ingress-nginx-controller-6598bff54d-sffqt -n ingress-nginx

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.7.0
  Build:         72ff21ed9e26cb969052c753633049ba8a87ecf9
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

W0415 05:07:19.264102       7 controller.go:279] ignoring ingress hm-kafka-kafka-bootstrap in hm-kafka based on annotation : ingress does not contain a valid IngressClass
I0415 05:07:19.264118       7 main.go:100] "successfully validated configuration, accepting" ingress="hm-kafka/hm-kafka-kafka-bootstrap"
W0415 05:07:19.264254       7 controller.go:279] ignoring ingress hm-kafka-kafka-0 in hm-kafka based on annotation : ingress does not contain a valid IngressClass
I0415 05:07:19.264262       7 main.go:100] "successfully validated configuration, accepting" ingress="hm-kafka/hm-kafka-kafka-0"
W0415 05:07:19.264344       7 controller.go:279] ignoring ingress hm-kafka-kafka-1 in hm-kafka based on annotation : ingress does not contain a valid IngressClass
W0415 05:07:19.264254       7 controller.go:279] ignoring ingress hm-kafka-kafka-2 in hm-kafka based on annotation : ingress does not contain a valid IngressClass
I0415 05:07:19.264351       7 main.go:100] "successfully validated configuration, accepting" ingress="hm-kafka/hm-kafka-kafka-1"
I0415 05:07:19.264354       7 main.go:100] "successfully validated configuration, accepting" ingress="hm-kafka/hm-kafka-kafka-2"
I0415 05:07:19.273788       7 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="hm-kafka/hm-kafka-kafka-1" error="ingress does not contain a valid IngressClass"
I0415 05:07:19.274943       7 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="hm-kafka/hm-kafka-kafka-2" error="ingress does not contain a valid IngressClass"
I0415 05:07:19.275020       7 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="hm-kafka/hm-kafka-kafka-0" error="ingress does not contain a valid IngressClass"
I0415 05:07:19.275129       7 store.go:429] "Ignoring ingress because of error while validating ingress class" ingress="hm-kafka/hm-kafka-kafka-bootstrap" error="ingress does not contain a valid IngressClass"

Based on this, the operator created the Ingress resources and then waits for the Ingress controller to confirm their address in the Status section. Right now the Ingress failed to validate the ingress class which is why Kafka brokers were not created.

Any guide to help fix would be appreciate, thanks!

Answer 1

I resolved by adding kubernetes.io/ingress.class: nginx annotation for Kafka bootstrap and brokers in the file my-kafka-persistent.yaml:

# ...
      - name: external
        port: 9094
        type: ingress
        tls: true
        configuration:
          bootstrap:
            host: kafka-bootstrap.192.168.1.149.nip.io
            annotations:
              kubernetes.io/ingress.class: nginx
          brokers:
          - broker: 0
            host: kafka-broker-0.192.168.1.149.nip.io
            annotations:
              kubernetes.io/ingress.class: nginx
          - broker: 1
            host: kafka-broker-1.192.168.1.149.nip.io
            annotations:
              kubernetes.io/ingress.class: nginx
          - broker: 2
            host: kafka-broker-2.192.168.1.149.nip.io
            annotations:
              kubernetes.io/ingress.class: nginx

After updating the file and redeploying the Kafka cluster, the brokers (hm-kafka-kafka-0, hm-kafka-kafka-1, hm-kafka-kafka-2) showed up: