Search code examples
kubernetesapache-kafkaconfluent-platform

Not able to externally connect to confluent kafka cluster's brokers with confluent for kubernetes

I am trying to set up External connection with Confluent for Kubernetes. I've followed the documentation :

spec:
  listeners:
    external:
      externalAccess:
        type: loadBalancer
        loadBalancer:
          domain: my.domain.com

My config file is really similar to the confluent-platform.yaml from confluent-kubernetes-examples

It might be worth to mention two things :

  1. my.domain.com is not the domain where your Kubernetes cluster is running (See [1])
  2. I have deployed a kafka cluster with Kraft configuration (Similar to [2])

All seem to be good and working correctly :

$ k get svc | grep LoadBalancer
kafka-0-lb                   LoadBalancer   10.43.103.230    1.2.3.4   9092:31455/TCP                                                            18h
kafka-1-lb                   LoadBalancer   10.43.18.77      1.2.3.5   9092:32514/TCP                                                            18h
kafka-2-lb                   LoadBalancer   10.43.250.91     1.2.3.6   9092:31104/TCP                                                            18h
kafka-bootstrap-lb           LoadBalancer   10.43.37.127     1.2.3.7   9092:31317/TCP                                                            18h

And I have added the external IP to my dns resolver :

DNS name                 External IP
b0.my.domain.com         1.2.3.4
b1.my.domain.com         1.2.3.5
b2.my.domain.com         1.2.3.6
kafka.my.domain.com      1.2.3.7

But then, when I am trying to connect from a VM with kafka-cli, which is not inside kubernetes :

$ ./bin/kafka-topics.sh --list --bootstrap-server kafka.my.domain.com:9092

Error while executing topic command : Timed out waiting for a node assignment. Call: listTopics
[2024-07-05 11:33:21,219] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
 (org.apache.kafka.tools.TopicCommand)

I have a TimedOut error. I tried a netcat command :

nc -zv kafka.my.domain.com 9092
Connection to kafka.my.domain.com 9092 port [tcp/*] succeeded!

When I try it inside any pod :

kubectl exec -it kafka-0 -c kafka -n confluent  -- /bin/bash

$ kafka-topics --list --bootstrap-server kafka.my.domain.com:9092

__consumer_offsets
_confluent-command
_confluent-controlcenter-7-6-0-0-AlertHistoryStore-changelog
...

It works fine.

What did I do wrong?

EDIT:

After a lot of investigation, here my new discoveries and precisions :

I installed kcat and I managed to query the metadata of the cluster with the command :

kafkacat -b kafka.my.domain.com -L

but when I tried to consume a topic, it didn't work :

% Auto-selecting Consumer mode (use -P or -C to override)
%7|1720456580.257|BRKMAIN|rdkafka#consumer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1720456580.257|STATE|rdkafka#consumer-1| [thrd::0/internal]: :0/internal: Broker changed state INIT -> UP
%7|1720456580.257|BROKER|rdkafka#consumer-1| [thrd:app]: kafka.my.domain.com:9092/bootstrap: Added new broker with NodeId -1
%7|1720456580.257|BRKMAIN|rdkafka#consumer-1| [thrd:kafka.my.domain.com:9092/bootstrap]: kafka.my.domain.com:9092/bootstrap: Enter main broker thread
%7|1720456580.257|CONNECT|rdkafka#consumer-1| [thrd:kafka.my.domain.com:9092/bootstrap]: kafka.my.domain.com:9092/bootstrap: broker in state INIT connecting
%7|1720456580.258|CONNECT|rdkafka#consumer-1| [thrd:kafka.my.domain.com:9092/bootstrap]: kafka.my.domain.com:9092/bootstrap: Connecting to ipv4#10.126.179.175:9092 (plaintext) with socket 7
%7|1720456580.258|STATE|rdkafka#consumer-1| [thrd:kafka.my.domain.com:9092/bootstrap]: kafka.my.domain.com:9092/bootstrap: Broker changed state INIT -> CONNECT
%7|1720456580.258|CONNECT|rdkafka#consumer-1| [thrd:kafka.my.domain.com:9092/bootstrap]: kafka.my.domain.com:9092/bootstrap: Connected to ipv4#10.126.179.175:9092
%7|1720456580.258|CONNECTED|rdkafka#consumer-1| [thrd:kafka.my.domain.com:9092/bootstrap]: kafka.my.domain.com:9092/bootstrap: Connected (#1)
%7|1720456580.258|FEATURE|rdkafka#consumer-1| [thrd:kafka.my.domain.com:9092/bootstrap]: kafka.my.domain.com:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1720456580.258|STATE|rdkafka#consumer-1| [thrd:kafka.my.domain.com:9092/bootstrap]: kafka.my.domain.com:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1720456580.270|FEATURE|rdkafka#consumer-1| [thrd:kafka.my.domain.com:9092/bootstrap]: kafka.my.domain.com:9092/bootstrap: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2
%7|1720456580.270|STATE|rdkafka#consumer-1| [thrd:kafka.my.domain.com:9092/bootstrap]: kafka.my.domain.com:9092/bootstrap: Broker changed state APIVERSION_QUERY -> UP
%7|1720456580.298|BROKER|rdkafka#consumer-1| [thrd:main]: b0.my.domain.com:9092/0: Added new broker with NodeId 0
%7|1720456580.298|BROKER|rdkafka#consumer-1| [thrd:main]: b1.my.domain.com:9092/1: Added new broker with NodeId 1
%7|1720456580.298|BROKER|rdkafka#consumer-1| [thrd:main]: b2.my.domain.com:9092/2: Added new broker with NodeId 2
%7|1720456580.298|CLUSTERID|rdkafka#consumer-1| [thrd:main]: kafka.my.domain.com:9092/bootstrap: ClusterId update "" -> "8becb4b2-33b5-444f-beQ"
%7|1720456580.302|BRKMAIN|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Enter main broker thread
%7|1720456580.303|CONNECT|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: broker in state INIT connecting
%7|1720456580.303|BRKMAIN|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Enter main broker thread
%7|1720456580.303|CONNECT|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: broker in state INIT connecting
%7|1720456580.303|BRKMAIN|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Enter main broker thread
%7|1720456580.303|CONNECT|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: broker in state INIT connecting
%7|1720456580.304|CONNECT|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Connecting to ipv4#10.126.179.174:9092 (plaintext) with socket 15
%7|1720456580.304|CONNECT|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Connecting to ipv4#10.126.179.173:9092 (plaintext) with socket 14
%7|1720456580.304|STATE|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Broker changed state INIT -> CONNECT
%7|1720456580.304|CONNECT|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Connected to ipv4#10.126.179.174:9092
%7|1720456580.304|CONNECTED|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Connected (#1)
%7|1720456580.304|STATE|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Broker changed state INIT -> CONNECT
%7|1720456580.304|FEATURE|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1720456580.304|CONNECT|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Connecting to ipv4#10.126.179.171:9092 (plaintext) with socket 16
%7|1720456580.304|STATE|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1720456580.304|STATE|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Broker changed state INIT -> CONNECT
%7|1720456580.304|CONNECT|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Connected to ipv4#10.126.179.171:9092
%7|1720456580.304|CONNECTED|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Connected (#1)
%7|1720456580.304|FEATURE|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1720456580.304|STATE|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1720456580.304|TOPBRK|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Topic transactions [0]: joining broker (rktp 0x7fbb1c002c50)
%7|1720456580.304|CONNECT|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Connected to ipv4#10.126.179.173:9092
%7|1720456580.304|CONNECTED|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Connected (#1)
%7|1720456580.304|FEATURE|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1720456580.304|STATE|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1720456583.379|BROKERFAIL|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: failed: err: Local: Broker transport failure: (errno: Operation now in progress)
%7|1720456583.379|FEATURE|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Updated enabled protocol features -ApiVersion to 
%7|1720456583.379|STATE|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Broker changed state APIVERSION_QUERY -> DOWN
%7|1720456583.379|REQERR|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: ApiVersionRequest failed: Local: Broker transport failure: explicit actions 0x0
%7|1720456583.379|BROKERFAIL|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: failed: err: Local: Not implemented: (errno: Operation now in progress)
%7|1720456583.379|FAIL|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: ApiVersionRequest failed: Local: Broker transport failure: probably due to old broker version
%7|1720456583.379|BROKERFAIL|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: failed: err: Local: Broker transport failure: (errno: Operation now in progress)
%7|1720456583.379|FEATURE|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Updated enabled protocol features -ApiVersion to 
%7|1720456583.379|STATE|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Broker changed state APIVERSION_QUERY -> DOWN
%7|1720456583.379|REQERR|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: ApiVersionRequest failed: Local: Broker transport failure: explicit actions 0x0
%7|1720456583.379|BROKERFAIL|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: failed: err: Local: Not implemented: (errno: Operation now in progress)
%7|1720456583.379|FAIL|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: ApiVersionRequest failed: Local: Broker transport failure: probably due to old broker version
%7|1720456583.379|BROKERFAIL|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: failed: err: Local: Broker transport failure: (errno: Operation now in progress)
%7|1720456583.379|FEATURE|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Updated enabled protocol features -ApiVersion to 
%7|1720456583.379|STATE|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Broker changed state APIVERSION_QUERY -> DOWN
%7|1720456583.379|REQERR|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: ApiVersionRequest failed: Local: Broker transport failure: explicit actions 0x0
%7|1720456583.379|BROKERFAIL|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: failed: err: Local: Not implemented: (errno: Operation now in progress)
%7|1720456583.379|FAIL|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: ApiVersionRequest failed: Local: Broker transport failure: probably due to old broker version
%7|1720456584.378|CONNECT|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: broker in state DOWN connecting
%7|1720456584.378|CONNECT|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: broker in state DOWN connecting
%7|1720456584.379|CONNECT|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: broker in state DOWN connecting
%7|1720456584.380|CONNECT|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Connecting to ipv4#10.126.179.173:9092 (plaintext) with socket 14
%7|1720456584.380|CONNECT|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Connecting to ipv4#10.126.179.174:9092 (plaintext) with socket 15
%7|1720456584.380|STATE|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Broker changed state DOWN -> CONNECT
%7|1720456584.380|CONNECT|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Connected to ipv4#10.126.179.173:9092
%7|1720456584.380|CONNECTED|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Connected (#2)
%7|1720456584.380|STATE|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Broker changed state DOWN -> CONNECT
%7|1720456584.380|APIVERSION|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Using (configuration fallback) 0.9.0 protocol features
%7|1720456584.380|CONNECT|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Connected to ipv4#10.126.179.174:9092
%7|1720456584.380|FEATURE|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Updated enabled protocol features to BrokerBalancedConsumer,ThrottleTime,Sasl,BrokerGroupCoordinator,LZ4
%7|1720456584.380|STATE|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Broker changed state CONNECT -> UP
%7|1720456584.380|CONNECTED|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Connected (#2)
%7|1720456584.380|APIVERSION|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Using (configuration fallback) 0.9.0 protocol features
%7|1720456584.380|FEATURE|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Updated enabled protocol features to BrokerBalancedConsumer,ThrottleTime,Sasl,BrokerGroupCoordinator,LZ4
%7|1720456584.380|STATE|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Broker changed state CONNECT -> UP
%7|1720456584.381|CONNECT|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Connecting to ipv4#10.126.179.171:9092 (plaintext) with socket 16
%7|1720456584.381|STATE|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Broker changed state DOWN -> CONNECT
%7|1720456584.381|CONNECT|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Connected to ipv4#10.126.179.171:9092
%7|1720456584.381|CONNECTED|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Connected (#2)
%7|1720456584.381|APIVERSION|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Using (configuration fallback) 0.9.0 protocol features
%7|1720456584.381|FEATURE|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Updated enabled protocol features to BrokerBalancedConsumer,ThrottleTime,Sasl,BrokerGroupCoordinator,LZ4
%7|1720456584.381|STATE|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Broker changed state CONNECT -> UP
^C%7|1720456584.602|DESTROY|rdkafka#consumer-1| [thrd:app]: Terminating instance
%7|1720456584.602|DESTROY|rdkafka#consumer-1| [thrd:main]: Destroy internal
%7|1720456584.602|DESTROY|rdkafka#consumer-1| [thrd:main]: Removing all topics
%7|1720456584.602|TOPBRK|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Topic transactions [0]: leaving broker (0 messages in xmitq, next leader (none), rktp 0x7fbb1c002c50)
%7|1720456584.602|TOPBRK|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Topic transactions [0]: no next leader, failing 0 message(s) in partition queue
%7|1720456584.602|TERMINATE|rdkafka#consumer-1| [thrd:kafka.my.domain.com:9092/bootstrap]: kafka.my.domain.com:9092/bootstrap: Handle is terminating: failed 0 request(s) in retry+outbuf
%7|1720456584.603|BROKERFAIL|rdkafka#consumer-1| [thrd:kafka.my.domain.com:9092/bootstrap]: kafka.my.domain.com:9092/bootstrap: failed: err: Local: Broker handle destroyed: (errno: Interrupted system call)
%7|1720456584.602|TERM|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Received TERMINATE op in state UP: 2 refcnts, 0 toppar(s), 0 fetch toppar(s), 0 outbufs, 1 waitresps, 0 retrybufs
%7|1720456584.603|STATE|rdkafka#consumer-1| [thrd:kafka.my.domain.com:9092/bootstrap]: kafka.my.domain.com:9092/bootstrap: Broker changed state UP -> DOWN
%7|1720456584.658|TERMINATE|rdkafka#consumer-1| [thrd::0/internal]: :0/internal: Handle is terminating: failed 0 request(s) in retry+outbuf
%7|1720456584.658|BROKERFAIL|rdkafka#consumer-1| [thrd::0/internal]: :0/internal: failed: err: Local: Broker handle destroyed: (errno: Success)
%7|1720456584.658|STATE|rdkafka#consumer-1| [thrd::0/internal]: :0/internal: Broker changed state UP -> DOWN
%7|1720456584.703|TERMINATE|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Handle is terminating: failed 0 request(s) in retry+outbuf
%7|1720456584.703|BROKERFAIL|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: failed: err: Local: Broker handle destroyed: (errno: Operation now in progress)
%7|1720456584.703|TERMINATE|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Handle is terminating: failed 0 request(s) in retry+outbuf
%7|1720456584.703|BROKERFAIL|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: failed: err: Local: Broker handle destroyed: (errno: Operation now in progress)
%7|1720456584.703|STATE|rdkafka#consumer-1| [thrd:b0.my.domain.com:9092/0]: b0.my.domain.com:9092/0: Broker changed state UP -> DOWN
%7|1720456584.703|STATE|rdkafka#consumer-1| [thrd:b2.my.domain.com:9092/2]: b2.my.domain.com:9092/2: Broker changed state UP -> DOWN
%7|1720456586.451|BROKERFAIL|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: failed: err: Local: Broker transport failure: (errno: Operation now in progress)
%7|1720456586.451|STATE|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Broker changed state UP -> DOWN
%7|1720456586.551|TERMINATE|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: Handle is terminating: failed 0 request(s) in retry+outbuf
%7|1720456586.551|BROKERFAIL|rdkafka#consumer-1| [thrd:b1.my.domain.com:9092/1]: b1.my.domain.com:9092/1: failed: err: Local: Broker handle destroyed: (errno: Operation now in progress)

Also, here is the kafka.properties that I found in /opt/confluent/ :

# kafka.properties
advertised.listeners=EXTERNAL://b0.my.domain.com:9092,INTERNAL://kafka-0.kafka.confluent.svc.cluster.local:9071,REPLICATION://kafka-0.kafka.confluent.svc.cluster.local:9072
broker.id=0
broker.rack=0
auto.create.topics.enable=false
config.providers=file
config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider
confluent.balancer.enable=true
confluent.license.topic.replication.factor=3
confluent.metrics.reporter.bootstrap.servers=kafka.confluent.svc.cluster.local:9071
confluent.metrics.reporter.publish.ms=30000
confluent.metrics.reporter.security.protocol=PLAINTEXT
confluent.metrics.reporter.topic.replicas=3
confluent.operator.managed=true
controller.listener.names=CONTROLLER
controller.quorum.voters=9990@kraftcontroller-0.kraftcontroller.confluent.svc.cluster.local:9074,9991@kraftcontroller-1.kraftcontroller.confluent.svc.cluster.local:9074,9992@kraftcontroller-2.kraftcontroller.confluent.svc.cluster.local:9074
default.replication.factor=3
delete.topic.enable=true
group.max.session.timeout.ms=1200000
inter.broker.listener.name=REPLICATION
inter.broker.protocol.version=3.4
kafka.rest.enable=true
listener.security.protocol.map=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT,REPLICATION:PLAINTEXT
listeners=EXTERNAL://:9092,INTERNAL://:9071,REPLICATION://:9072
log.dirs=/mnt/data/data0/logs
log.message.format.version=3.4
log.retention.check.interval.ms=300000
log.retention.hours=168
log.segment.bytes=1073741824
metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter
min.insync.replicas=2
num.io.threads=8
num.network.threads=4
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.commit.timeout.ms=15000
offsets.retention.minutes=10080
offsets.topic.compression.codec=3
offsets.topic.replication.factor=3
process.roles=broker
replica.lag.time.max.ms=45000
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
transaction.state.log.min.isr=2
transaction.state.log.replication.factor=3
Solution

  • I finally managed to figure out my problem:

    For LoadBalancer IP attribution, I am using MetalLB. However, the kube-proxy was not colocated on the Kafka nodes. By default, Kafka LoadBalancer listeners use the externalTrafficPolicy=Local policy, which can only redirect traffic to a pod on the same node as the kube-proxy. Since the kube-proxy was not present on the Kafka nodes, the packets were dropped. The solution was to change the configuration as follows:

    listeners:
  external:
    externalAccess:
      type: loadBalancer
      loadBalancer:
        domain: ## Domain 
        externalTrafficPolicy: Cluster # <-- HERE

    This change ensures that traffic is routed correctly within the cluster, regardless of the node where the kube-proxy is running.