Search code examples
debeziumksqldbapicurio-registry

ksqlDB with Apicurio Registry

I’m trying to make ksqlDB work with Apicurio Registry (RedHat) and have had a few issues with compatibility with Debezium.

So, Apicurio has 4 different APIs that can be used. The “official” and recommended api is called "/registry/v2" and it seems to work fine with my Debezium connector. ksqlDB however, when configured to use it, is trying to access “/subjects” which isn’t known by this API. For that to work, I have to point it to the "/ccompat/v6" API which is made to be compatible with Confluent Schema Registry clients. When I make ksqlDB point to this one, it works all good… until the IDs (Global-Id and Content-Id) in Apicurio aren’t synced anymore.

As seen here, Apicurio has 3 different IDs to identifiy a schema: enter image description here

Debezium uses the “Global-Id” to put in the as “Schema-Id” of each Kafka messages it produces. ksqlDB then reads that ID and tries to find it in Apicurio, but using the “Content-Id” instead. So either it’ll never find it or it will find a random schema. In both cases, it gives me a deserialization error :

ERROR {“type”:0,“deserializationError”:{“target”:“value”,“errorMessage”:“Error deserializing message from topic: debezium-topic-1”,“recordB64”:null,“cause”:["Failed to deserialize data for topic debezium-topic-1 to Avro: ",“Error retrieving Avro value schema for id 96”,“No content with id/hash ‘contentId-96’ was found.; error code: 40403”]

Is there a way to make ksqlDB use the official Apicurio Registry API (/registry/v2/) or alternatively, is there a way to force it to use “Global-Id” instead of “Content-Id” ?

Thanks for the help!

Solution

  • Found my answer... there is a "Legacy ID mode" in Apicurio which you can enable in the settings through the Apicurio web UI :

    enter image description here

    Once I turned that on, it worked perfectly with ksqlDB and the /ccompat/v6 API.