Search code examples
apache-kafkakafka-topic

How to get Load of my Kafka Topic. Is there any API with which I can measure the load on my Kafka Topic?


I have a Kafka topic on my spring boot application on which I am sending some data from producer. I want to check the load of my topic so that I can create new topics, if load on the previous topic exceeds


Solution

  • Topics don't have "load" in the traditional sense. Sure, you can use JMX metrics to measure incoming byte rates, but that is network load, as measured by the broker. You can also measure outgoing rates by the producer, per partition, and aggregate to get data by topic.

    The brokers hosting the partitions do have load; measurable network, disk , and CPU load.

    Secondly, your producers would all need to be updated to actually send data to those new topics you'd created; they nor the brokers would know to "distribute load" to them.

    The correct way to reduce broker load, and distribute data to more brokers is to increase the cluster size, and the correct way to scale production is to add more partitions.