Search code examples
linuxapache-kafkaapache-zookeeper

Apache Kafka + how to defected Kafka clusters with Kraft mode or Kafka with zookeeper mode

We have dozens different Kafka clusters labs

Some of them are with Kraft mode and most of them are ordinary Kafka cluster with zookeeper

Since we run smoke tests scripts based on python/perl/bash on all Kafka cluster

then we want identify which mode Kafka is running - Kraft or not Kraft

I will share here some options , but I must to say that all options that I tested still not convincing and we want to find better option

The first option , is by using the Kafka PID as

ps -ef | grep ` netstat -tulpn | grep "9092" | awk '{print $NF}' | sed s'/\// /g' | awk '{print $1}' ` | grep -v grep | grep -i kraft

This gives the PID content , but if binary/s folder/s not include the name Kraft then , we not filter the string - Kraft

Second option is to use the Kafka cli - kafka-metadata-quorum.sh

bash kafka-metadata-quorum.sh  --bootstrap-server `hostname`:9092 describe --status
ClusterId:              uuZQoN9-TGmVU3lKDFCtww
LeaderId:               2001
LeaderEpoch:            975
HighWatermark:          224514
MaxFollowerLag:         0
MaxFollowerLagTimeMs:   0
CurrentVoters:          [2001,2002,2003]
CurrentObservers:       [1010,1001,1002,1003]

in Kraft mode we should get the above details , and in zookeeper mode we get the following

kafka-metadata-quorum.sh  --bootstrap-server `hostname`:9092 describe --status
org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support DESCRIBE_QUORUM
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support DESCRIBE_QUORUM
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
        at org.apache.kafka.tools.MetadataQuorumCommand.handleDescribeStatus(MetadataQuorumCommand.java:208)
        at org.apache.kafka.tools.MetadataQuorumCommand.execute(MetadataQuorumCommand.java:108)
        at org.apache.kafka.tools.MetadataQuorumCommand.mainNoExit(MetadataQuorumCommand.java:61)
        at org.apache.kafka.tools.MetadataQuorumCommand.main(MetadataQuorumCommand.java:56)
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support DESCRIBE_QUORUM

I also thinking about option as to create flag as file under /var/lib/kafka/kraft ,

but I don't like this since files can deleted by users

Other last option is defected controller PID ( Process ID ) , so if its running then we can say Kafka is Kraft mode , but actually this option not so good because its relevant for combine mode ( when broker and controller running on same host ) and not support separated clusters when controller are separated on dedicated hosts , and also if controller is down from some reason then its a problem

Solution

  • jps might include the work Kraft; cannot recall.

    From the cluster itself, the best option would be to parse the server.properties file for the existence of quorum.voters or lack of zookeeper.connect for Kraft mode.

    Otherwise, kafka-metadata-quorum.sh should work too, since non-Kraft brokers should not respond to this.