Search code examples
kubernetesapache-kafkabitnami

Bitnami Kafka for Kibernetes - java.net.UnknownHostException: kafka-0.kafka-headless.kafka.svc.cluster.local

I have installed bitnami kafka on my k8 cluster. But I continuously see error java.net.UnknownHostException: kafka-0.kafka-headless.kafka.svc.cluster.local in kafka-o logs.

I login to the pod (kafka-o) and ran nslookup for kafka-0.kafka-headless.kafka.svc.cluster.local but the IP gets resolve with no issues

root@kafka-0:/# nslookup  kafka-0.kafka-headless.kafka.svc.cluster.local
Server:         10.43.0.10
Address:        10.43.0.10#53

Name:   kafka-0.kafka-headless.kafka.svc.mosip02.local
Address: 10.42.4.123

The netstat log from kafka-0 pod

root@kafka-0:/# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:35267           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:9092            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:9093            0.0.0.0:*               LISTEN      -

Error stack from kubectl -n kafka logs kafka-0

[2021-07-26 07:59:29,350] WARN [Controller id=0, targetBrokerId=0] Error connecting to node kafka-0.kafka-headless.kafka.svc.cluster.local:9093 (id: 0 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: kafka-0.kafka-headless.kafka.svc.cluster.local
        at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
        at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
        at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
        at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:111)
        at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:512)
        at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:466)
        at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:172)
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:985)
        at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:311)
        at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:65)
        at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:292)
        at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:246)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Solution

  • Give a try to edit the clusterDomain parameter setting it to mosip02.local instead of cluster.local when deploying the chart