We recently started migrating our test clusters from Zookeeper to KRaft, all running Confluent Kafka on prem. In KRaft, cluster's metadata is stored in __cluster_metadata topic (see: https://developer.confluent.io/courses/architecture/control-plane/#kraft-cluster-metadata) . What I noticed after the first cluster was migrated is that I am not able to describe this topic using kafka-topics command, it just does not exist:
Error while executing topic command : Topic '__cluster_metadata' does not exist as expected
Yet, the topic's directory is found in the broker:
sh-4.4$ cd /mnt/data/data0/logs
sh-4.4$ ls -dl __cluster_metadata-0
drwxr-sr-x. 2 1002150000 1002150000 4096 Jul 25 08:36 __cluster_metadata-0
Confluent's support replied to this issue saying "it is not an actual internal kafka topic".
If you run Kafka with KRaft, can you actually check if __cluster_metadata topic can be described with kafka-topics command? Not that I would need this on a daily basis, I just want to check what __cluster_metadata really is.
__cluster_metadata is not a topic. As the name indicates, this is used to store the cluster metadata in KRaft mode. This data used to be stored in ZooKeeper.
As it's not a topic you cannot list or describe it, and cannot produce to or consume from it either. The cluster metadata contains details about the brokers, topics, configurations, etc.
All the information useful for users is exposed via the regular APIs (either the tools in /bin, or the Admin API. So you should not have to interact with the cluster metadata directly, like you did not have to look into ZooKeeper when you used to run Kafka in ZooKeeper mode.
If you are curious, you can use the kafka-metadata-shell.sh tool to explore the metadata content. For example:
./bin/kafka-metadata-shell.sh --snapshot /tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log
Loading...
Starting...
[ Kafka Metadata Shell ]
>> ls /
image local
>> ls /image/cluster/brokers
1
>> cat /image/cluster/brokers/1
BrokerRegistration(id=1, epoch=6, incarnationId=hNHfnBOkTtmJVS7bE7QoJQ, listeners=[Endpoint(listenerName='PLAINTEXT', securityProtocol=PLAINTEXT, host='localhost', port=9092)], supportedFeatures={metadata.version: 1-20}, rack=Optional.empty, fenced=true, inControlledShutdown=true, isMigratingZkBroker=false, directories=[oujsN3ZZR5ChA3cSj-E92w])
You can use the help command to see all the usable commands.
Note that the broker owning the file must be stopped, otherwise the tool will fail with:
Unexpected error: Unable to lock /tmp/kraft-combined-logs. Please ensure that no broker or controller process is using this directory before proceeding.