Strimzi Kafka setup with GKE internal loadbalancer

Question

Followed https://strimzi.io/quickstarts/ and https://strimzi.io/blog/2019/05/13/accessing-kafka-part-4/ to use GKE internal loadbalancer with Strimzi. After adding the internal load balancer Strimzi provisioned two loadbalancer service with external IP.

Kafka % k get svc -n kafka
NAME                                  TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                               AGE
my-cluster-kafka-0                    LoadBalancer   xx.xxx.xx.xxx   bb.bb.bbb.bb   9094:30473/TCP                        3d1h
my-cluster-kafka-bootstrap            ClusterIP      xx.xxx.xx.xxx   <none>         9091/TCP,9092/TCP,9093/TCP            25d
my-cluster-kafka-brokers              ClusterIP      None            <none>         9090/TCP,9091/TCP,9092/TCP,9093/TCP   25d
my-cluster-kafka-external-bootstrap   LoadBalancer   xx.xxx.xx.xxx   aa.aa.aaa.aa   9094:30002/TCP                        3d1h
my-cluster-zookeeper-client           ClusterIP      xx.xxx.xx.xxx   <none>         2181/TCP                              25d
my-cluster-zookeeper-nodes            ClusterIP      None            <none>         2181/TCP,2888/TCP,3888/TCP            25d

The producer/consumer flow is working inside the cluster using my-cluster-kafka-bootstrap and I can also curl the my-cluster-kafka-external-bootstrap addess aa.aa.aaa.aa:9094 from outside the cluster. However after producing to aa.aa.aaa.aa:9094 from outside the cluster my producer logged the error below.

Connection to node 0 (bb.bb.bbb.bb:9094) could not be established. Broker may not be available.

which seem to indicate my-cluster-kafka-external-bootstrap is forwarding the traffic to my-cluster-kafka-0. And per kubectl get svc -o yaml output only my-cluster-kafka-external-bootstrap was setup as a GKE internal LB. Since there are various firewall rules in our enviroment I suspect that my-cluster-kafka-0 needs to be set up as a GKE internal LB as well for the producer to work. Does this seem to be the issue? How do I update Strimzi to make both LB internal? Thanks.

A relevant question before Strimzi kafka accessing it privately with in GKE. But it didn't help after I turn off tls.

Answer 1

Answering own question. Appearntly Strimzi provision one LB per broker which is the my-cluster-kafka-0 here. The listener config can specify these per broker LBs like this https://strimzi.io/blog/2019/05/13/accessing-kafka-part-4/

# ...
listeners:
  # ...
  - name: external
    port: 9094
    type: loadbalancer
    tls: true
    authentication:
      type: tls
    configuration:
      bootstrap:
        annotations:
          service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
      brokers:
      - broker: 0
        annotations:
          service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
      - broker: 1
        annotations:
          service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
      - broker: 2
        annotations:
          service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
# ...