I'm building a Kafka Streams application using Protobuf for message schemas. For now the application itself is just piping from one topic to another. I'm running Kafka locally using the Confluent platform all-in-one docker-compose file.
One of my schemas (foo.proto
) uses a Struct
field, so prior to starting my app I have registered both foo.proto
and struct.proto
on the schema registry.
When I start my app the protobuf serializer runs a method called resolveDependencies
, leading it to re-register subtruct.proto
. The (local) schema registry returns a 409 with message:
Schema being registered is incompatible with an earlier schema
The code that originally registers struct.proto
and the app are both using version 3.12
of the protobuf-java
library.
Looking at the docker logs for the registry, they give me a bit more detail with the following message:
Found incompatible change: Difference{fullPath='#/ListValue/1', type=FIELD_NAMED_TYPE_CHANGED}
Looking at that particular field the already registered schema has the following definition
message ListValue {
repeated Value values = 1;
}
and the schema my application is trying to register has
message ListValue {
repeated .google.protobuf.Value values = 1;
}
My questions
The solution in my case was just to not pre-register the schemas, and instead start from a clean schema registry. The kafka-streams app auto-registered the relevant schemas.
I am guessing that the way I registered the original schema wasn't quite correct.