Search code examples
apache-kafkakafka-producer-apiconfluent-platformconfluent-schema-registry

Metadata information from kafka

I am new to Confluent/Kafka and I want to find metadata information from kafka

I want to know

  1. list of producers
  2. list of topics
  3. schema information for topic

Confluent version is 5.0

What are classes (methods) that can give this information?
Are there any Rest API's for the same
Also is zookeeper connection necessary to get this information.

Solution

  • 1) I don't think that Kafka brokers are aware of producers that produce messages in topics and therefore there is no command line tool for listing them. However, an answer to this SO question suggests that you can list producers by viewing the MBeans over JMX.


    2) In order to list the topics you need to run:

    kafka-topics --zookeeper localhost:2181 --list

    Otherwise, if you want to list the topics using a Java client, you can call listTopics() method of KafkaConsumer.

    You can also fetch the list of topics through ZooKeeper

    ZkClient zkClient = new ZkClient("zkHost:zkPort");
List<String> topics = JavaConversions.asJavaList(ZkUtils.getAllTopics(zkClient));


    3) To get the schema information for a topic you can use Schema Registry API

    In particular, you can fetch all subjects by calling:

    GET /subjects HTTP/1.1
Host: schemaregistry.example.com
Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json

    which should give a response similar to the one below: 

    HTTP/1.1 200 OK
Content-Type: application/vnd.schemaregistry.v1+json

["subject1", "subject2"]

    You can then get all the versions of a particular subject: 

    GET /subjects/subject-name/versions HTTP/1.1
Host: schemaregistry.example.com
Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json

    And finally, you can get a specific version of the schema registered under this subject

    GET /subjects/subject_name/versions/1 HTTP/1.1
Host: schemaregistry.example.com
Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json

    Or just the latest registered schema: 

    GET /subjects/subject-name/versions/latest HTTP/1.1
Host: schemaregistry.example.com
Accept: application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json

    In order to perform such actions in Java, you can either prepare your own GET requests (see how to do it here) or use Confluent's Schema Registry Java Client. You can see the implementation and the available methods in their Github repo.


    Regarding your question about Zookeeper, note that ZK is a requirement for Kafka.

    Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one. You can use the convenience script packaged with kafka to get a quick-and-dirty single-node ZooKeeper instance.