Search code examples
scalaapache-kafkaapache-zookeeper

Basic kafka topic availability checker

I need a simple health checker for Apache Kafka. I dont want something large and complex like Yahoo Kafka Manager, basically I want to check if a topic is healthy or not and if a consumer is healthy.

My first idea was to create a separate heart-beat topic and periodically send and read messages to/from it in order to check availability and latency.

The second idea is to read all the data from Apache Zookeeper. I can get all brokers, partitions, topics etc. from ZK, but I dont know if ZK can provide something like failure detection info.

As I said, I need something simple that I can use in my app health checker.

Solution

  • Some existing tools you can try them out if you haven't yet -

    Burrow Linkedin's Kafka Consumer Lag Checking

    exhibitor Netflix's ZooKeeper co-process for instance monitoring, backup/recovery, cleanup and visualization.

    Kafka System Tools Kafka command line tools