Search code examples
ksqldb

How can I config ksqldb to understand binary proto-buffer message in kafka?

I want to use ksqldb to do some query on the streamed data encoded in proto-buffer format. But I don't have a clue how to achieve it. What if the binary message data is plain c struct, how to decode the c-struct messages and apply queries on the stream data?

Solution

  • ksqlDB supports Protobuf that's been serialised using the Schema Registry format. To specify your data as protobuf use FORMAT='PROTOBUF' e.g.

    CREATE STREAM my_stream
  WITH (KAFKA_TOPIC='my_topic', 
        FORMAT='PROTOBUF');

    The schema itself is fetched from the Schema Registry.

    For more details see https://docs.ksqldb.io/en/latest/reference/serialization/