Search code examples
phpapache-kafkaprotocol-buffersconfluent-schema-registry

How to serialize protobuf message with schema id


Looking for a library or algorithm that implements serialization of a message in protobuf format with schema version which retrieved from confluent schema registry. I use the php-rdkafka extension to send messages to Kafka topic.


Solution

  • The format of the messages is documented here

    For protobuf - it is a byte array of

    magic-byte, schema-id, message-indexes, protobuf-payload
    

    You can find source code here / here that generates the same wire-format for Avro using the Confluent Registry, and you'd need to do something very similar once you have raw bytearray for any protobuf object.

    And that library is what is used by php-rdkafka-examples repo