I am trying to setup Kafka in my K8S. For that, I am using Strimzi, with this configuration file :
apiVersion: v1
kind: PersistentVolume
metadata:
name: logs-kafka-worker-01-volume
labels:
type: local
spec:
storageClassName: logs-kafka-worker-01
capacity:
storage: 1000Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/tmp/k8s-logs-kafka"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-eu-de-worker-01
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: logs-kafka-worker-02-volume
labels:
type: local
spec:
storageClassName: logs-kafka-worker-02
capacity:
storage: 1000Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/tmp/k8s-logs-kafka"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-eu-de-worker-02
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: logs-kafka-worker-03-volume
labels:
type: local
spec:
storageClassName: logs-kafka-worker-03
capacity:
storage: 1000Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/tmp/k8s-logs-kafka"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-eu-de-worker-03
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: logs-zookeeper-worker-01-volume
labels:
type: local
spec:
storageClassName: logs-zookeeper-worker-01
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/tmp/k8s-logs-zookeeper"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-eu-de-worker-01
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: logs-zookeeper-worker-02-volume
labels:
type: local
spec:
storageClassName: logs-zookeeper-worker-02
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/tmp/k8s-logs-zookeeper"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-eu-de-worker-02
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: logs-zookeeper-worker-03-volume
labels:
type: local
spec:
storageClassName: logs-zookeeper-worker-03
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/tmp/k8s-logs-zookeeper"
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-eu-de-worker-03
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: logs
namespace: logs
spec:
kafka:
resources:
requests:
memory: 16Gi
cpu: "8"
limits:
memory: 24Gi
cpu: "12"
config:
log.retention.bytes: 536870912000
log.retention.hours: 24
log.retention.check.interval.ms: 600000
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
log.message.format.version: "2.8"
inter.broker.protocol.version: "2.8"
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
livenessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
readinessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
replicas: 3
storage:
deleteClaim: false
size: 1000Gi
type: persistent-claim
class: logs-kafka-worker
overrides:
- broker: 0
class: logs-kafka-worker-01
- broker: 1
class: logs-kafka-worker-02
- broker: 2
class: logs-kafka-worker-03
zookeeper:
resources:
requests:
memory: 2Gi
cpu: "1"
limits:
memory: 4Gi
cpu: "1"
replicas: 3
storage:
deleteClaim: false
size: 10Gi
type: persistent-claim
class: logs-zookeeper-worker
overrides:
- broker: 0
class: logs-zookeeper-worker-01
- broker: 1
class: logs-zookeeper-worker-02
- broker: 2
class: logs-zookeeper-worker-03
The ZooKeepers pods has been created and running state but the kafka pods are in CrashLoopBackOff state ...
And it looks like there is a timeout in connection between kafka and zookeeper, but that's strange the IP:Port is matching. Please find the logs and list of pods below.
Kafka :
2023-03-02 08:37:27,364 INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper) [main]
2023-03-02 08:37:27,364 INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper) [main]
2023-03-02 08:37:27,364 INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper) [main]
2023-03-02 08:37:27,364 INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper) [main]
2023-03-02 08:37:27,364 INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper) [main]
2023-03-02 08:37:27,364 INFO Client environment:os.version=5.10.0-21-amd64 (org.apache.zookeeper.ZooKeeper) [main]
2023-03-02 08:37:27,364 INFO Client environment:user.name=kafka (org.apache.zookeeper.ZooKeeper) [main]
2023-03-02 08:37:27,364 INFO Client environment:user.home=/home/kafka (org.apache.zookeeper.ZooKeeper) [main]
2023-03-02 08:37:27,364 INFO Client environment:user.dir=/opt/kafka (org.apache.zookeeper.ZooKeeper) [main]
2023-03-02 08:37:27,364 INFO Client environment:os.memory.free=5074MB (org.apache.zookeeper.ZooKeeper) [main]
2023-03-02 08:37:27,364 INFO Client environment:os.memory.max=5120MB (org.apache.zookeeper.ZooKeeper) [main]
2023-03-02 08:37:27,364 INFO Client environment:os.memory.total=5120MB (org.apache.zookeeper.ZooKeeper) [main]
2023-03-02 08:37:27,365 INFO Initiating client connection, connectString=logs-zookeeper-client:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@1c852c0f (org.apache.zookeeper.ZooKeeper) [main]
2023-03-02 08:37:27,394 INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket) [main]
2023-03-02 08:37:27,396 INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn) [main]
2023-03-02 08:37:27,397 INFO Starting poller (io.strimzi.kafka.agent.KafkaAgent) [main]
2023-03-02 08:37:27,398 INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient) [main]
2023-03-02 08:37:27,401 INFO Opening socket connection to server logs-zookeeper-client/10.233.20.91:2181. (org.apache.zookeeper.ClientCnxn) [main-SendThread(logs-zookeeper-client:2181)]
2023-03-02 08:37:27,615 INFO SSL handler added for channel: [id: 0x2616d54c] (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1]
2023-03-02 08:37:45,399 INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient) [main]
2023-03-02 08:37:45,456 WARN Client session timed out, have not heard from server in 18058ms for session id 0x0 (org.apache.zookeeper.ClientCnxn) [main-SendThread(logs-zookeeper-client:2181)]
2023-03-02 08:37:45,457 WARN An exception was thrown while closing send thread for session 0x0. (org.apache.zookeeper.ClientCnxn) [main-SendThread(logs-zookeeper-client:2181)]
org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 18058ms for session id 0x0
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1258)
2023-03-02 08:37:45,459 INFO channel is told closing (org.apache.zookeeper.ClientCnxnSocketNetty) [main]
2023-03-02 08:37:45,460 WARN future isn't success. (org.apache.zookeeper.ClientCnxnSocketNetty) [nioEventLoopGroup-2-1]
java.nio.channels.ClosedChannelException
at io.netty.channel.nio.AbstractNioChannel.doClose(AbstractNioChannel.java:502)
at io.netty.channel.socket.nio.NioSocketChannel.doClose(NioSocketChannel.java:349)
at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:754)
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:731)
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:620)
at io.netty.channel.nio.NioEventLoop.closeAll(NioEventLoop.java:772)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:529)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-03-02 08:37:45,461 INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper) [main]
2023-03-02 08:37:45,462 INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn) [main-EventThread]
2023-03-02 08:37:45,462 INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient) [main]
2023-03-02 08:37:45,464 ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) [main]
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:254)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:108)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:2262)
at kafka.zk.KafkaZkClient$.createZkClient(KafkaZkClient.scala:2354)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:593)
at kafka.server.KafkaServer.startup(KafkaServer.scala:213)
at kafka.Kafka$.main(Kafka.scala:115)
at kafka.Kafka.main(Kafka.scala)
2023-03-02 08:37:45,465 INFO shutting down (kafka.server.KafkaServer) [main]
2023-03-02 08:37:45,504 INFO App info kafka.server for 2 unregistered (org.apache.kafka.common.utils.AppInfoParser) [main]
2023-03-02 08:37:45,504 INFO shut down completed (kafka.server.KafkaServer) [main]
2023-03-02 08:37:45,504 ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$) [main]
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:254)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:108)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:2262)
at kafka.zk.KafkaZkClient$.createZkClient(KafkaZkClient.scala:2354)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:593)
at kafka.server.KafkaServer.startup(KafkaServer.scala:213)
at kafka.Kafka$.main(Kafka.scala:115)
at kafka.Kafka.main(Kafka.scala)
2023-03-02 08:37:45,507 INFO shutting down (kafka.server.KafkaServer) [kafka-shutdown-hook]
All Zookeeper pod logs are showing INFO Processing ruok command with no Errors
Zookeeper operator :
2023-03-02 08:37:25 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -2 (logs-kafka-1.logs-kafka-brokers.logs.svc.cluster.local/10.233.100.144:9091) could not be established. Broker may not be available.
2023-03-02 08:37:25 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -3 (logs-kafka-2.logs-kafka-brokers.logs.svc.cluster.local/10.233.77.144:9091) could not be established. Broker may not be available.
2023-03-02 08:37:26 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -1 (logs-kafka-0.logs-kafka-brokers.logs.svc.cluster.local/10.233.103.77:9091) could not be established. Broker may not be available.
2023-03-02 08:37:26 WARN KafkaAvailability:72 - Reconciliation #151(timer) Kafka(logs/logs): failed to get topic descriptions
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
at org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$4.handleFailure(KafkaAdminClient.java:1922) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$Call.handleTimeoutFailure(KafkaAdminClient.java:851) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:817) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$TimeoutProcessor.handleTimeouts(KafkaAdminClient.java:947) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.timeoutPendingCalls(KafkaAdminClient.java:1026) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1380) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1344) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
2023-03-02 08:37:26 WARN KafkaAvailability:90 - Reconciliation #151(timer) Kafka(logs/logs): Error determining whether it is safe to restart pod 1
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
at org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$4.handleFailure(KafkaAdminClient.java:1922) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$Call.handleTimeoutFailure(KafkaAdminClient.java:851) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:817) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$TimeoutProcessor.handleTimeouts(KafkaAdminClient.java:947) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.timeoutPendingCalls(KafkaAdminClient.java:1026) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1380) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1344) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
2023-03-02 08:37:35 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -1 (logs-kafka-0.logs-kafka-brokers.logs.svc.cluster.local/10.233.103.77:9091) could not be established. Broker may not be available.
2023-03-02 08:37:35 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -2 (logs-kafka-1.logs-kafka-brokers.logs.svc.cluster.local/10.233.100.144:9091) could not be established. Broker may not be available.
2023-03-02 08:37:36 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -3 (logs-kafka-2.logs-kafka-brokers.logs.svc.cluster.local/10.233.77.144:9091) could not be established. Broker may not be available.
2023-03-02 08:37:36 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -2 (logs-kafka-1.logs-kafka-brokers.logs.svc.cluster.local/10.233.100.144:9091) could not be established. Broker may not be available.
2023-03-02 08:37:36 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -1 (logs-kafka-0.logs-kafka-brokers.logs.svc.cluster.local/10.233.103.77:9091) could not be established. Broker may not be available.
2023-03-02 08:37:37 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -3 (logs-kafka-2.logs-kafka-brokers.logs.svc.cluster.local/10.233.77.144:9091) could not be established. Broker may not be available.
2023-03-02 08:37:37 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -2 (logs-kafka-1.logs-kafka-brokers.logs.svc.cluster.local/10.233.100.144:9091) could not be established. Broker may not be available.
2023-03-02 08:37:37 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -1 (logs-kafka-0.logs-kafka-brokers.logs.svc.cluster.local/10.233.103.77:9091) could not be established. Broker may not be available.
2023-03-02 08:37:38 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -3 (logs-kafka-2.logs-kafka-brokers.logs.svc.cluster.local/10.233.77.144:9091) could not be established. Broker may not be available.
2023-03-02 08:37:38 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -2 (logs-kafka-1.logs-kafka-brokers.logs.svc.cluster.local/10.233.100.144:9091) could not be established. Broker may not be available.
2023-03-02 08:37:39 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -1 (logs-kafka-0.logs-kafka-brokers.logs.svc.cluster.local/10.233.103.77:9091) could not be established. Broker may not be available.
2023-03-02 08:37:39 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -3 (logs-kafka-2.logs-kafka-brokers.logs.svc.cluster.local/10.233.77.144:9091) could not be established. Broker may not be available.
2023-03-02 08:37:39 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -2 (logs-kafka-1.logs-kafka-brokers.logs.svc.cluster.local/10.233.100.144:9091) could not be established. Broker may not be available.
2023-03-02 08:37:39 WARN NetworkClient:775 - [AdminClient clientId=adminclient-2] Connection to node -1 (logs-kafka-0.logs-kafka-brokers.logs.svc.cluster.local/10.233.103.77:9091) could not be established. Broker may not be available.
2023-03-02 08:37:47 WARN NetworkClient:997 - [AdminClient clientId=adminclient-2] Error connecting to node logs-kafka-2.logs-kafka-brokers.logs.svc.cluster.local:9091 (id: -3 rack: null)
java.net.UnknownHostException: logs-kafka-2.logs-kafka-brokers.logs.svc.cluster.local: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:933) ~[?:?]
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1534) ~[?:?]
at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852) ~[?:?]
at java.net.InetAddress.getAllByName0(InetAddress.java:1524) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1381) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1305) ~[?:?]
at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:510) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:467) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:990) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1141) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1401) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1344) ~[org.apache.kafka.kafka-clients-3.4.0.jar:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Pods list :
NAME READY STATUS RESTARTS AGE
pod/logs-kafka-0 0/1 CrashLoopBackOff 8 (26s ago) 19m
pod/logs-kafka-1 0/1 CrashLoopBackOff 8 (26s ago) 19m
pod/logs-kafka-2 0/1 CrashLoopBackOff 8 (15s ago) 19m
pod/logs-zookeeper-0 1/1 Running 0 19m
pod/logs-zookeeper-1 1/1 Running 0 19m
pod/logs-zookeeper-2 1/1 Running 0 19m
pod/strimzi-cluster-operator-677bf6869f-w9v6f 1/1 Running 0 16h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/logs-kafka-bootstrap ClusterIP 10.233.30.141 <none> 9091/TCP,9092/TCP,9093/TCP 19m
service/logs-kafka-brokers ClusterIP None <none> 9090/TCP,9091/TCP,9092/TCP,9093/TCP 19m
service/logs-zookeeper-client ClusterIP 10.233.20.91 <none> 2181/TCP 19m
service/logs-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 19m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/strimzi-cluster-operator 1/1 1 1 16h
NAME DESIRED CURRENT READY AGE
replicaset.apps/strimzi-cluster-operator-677bf6869f 1 1 1 16h
Someone already face this issue or have an idea to resolve it ? Thanks
The problem is due to Firewall on nodes servers, I forgot to set two rules :
iptables -t filter -A INPUT -d 10.0.0.0/8 -j ACCEPT
iptables -t filter -A INPUT -s 10.0.0.0/8 -j ACCEPT