Search code examples
apache-kafkaprotocol-bufferskcat

kcat protobuf deserialization

I'm using kcat to check the content of kafka topics when working locally but, when messages are serialized with protobuf, the result I get is an unreadable stream of encoded characters. I'm aware of the existence of some other kafka-consumers tools (Kafdrop, AKHQ, Kowl, Kadek...) but I'm looking for the simplest option which fits my needs.

Does kcat support protobuf key/value deserialization from protofile?
Is there any simple terminal-based tool which allows this?

Solution

  • I've had luck with this command:

    kcat -C -t <topic> -b <kafkahost>:9092 -o -1 -e -q -D "" | protoc --decode=<full message class> path/to/my.proto --proto_path <proto_parent_folder>

    Update 12/2023 - Here's a more streamlined command with better guidance.

    kcat -C -t <topic> -b <kafkahost>:9092 -o -1 -e -q -D "" | protoc --decode <package>.<Message Type> path/to/my.proto

    Where is the package defined in the .proto file and <Message Type> is the name of the Message.

    Ie.: user.proto:

    package some.package;

message User {
  string id = 1;
  string email = 2;
}

    The command would be:

    kcat -C -t <topic_name> -b host.docker.internal:9092 -o -1 -e -q -D "" | protoc --decode some.package.User user.proto