Search code examples
apache-kafkaapache-kafka-connectconfluent-schema-registrydebezium

Upgrading to Debezium Postgres Connector v2.0.1 causing schema incompatibility issues

I have an application that uses Kafka, Debezium Connector for Postgres, and a Schema registry. Currently I am running version 1.9.3 of Debezium Connector and everything runs fine.

Lately, when I tried upgrading the connector version to 2.0.1, I am facing some Schema incompatibility issues. Following is the error message -

io.confluent.rest.exceptions.RestException: Schema being registered is incompatible with an earlier schema for subject

I went through the release notes for Debezium Connector and strongly believes that this could be because of this breaking change.

All schemas used by Debezium are now defined in a central point, properly named and versioned (DBZ-4365, DBZ-5044). This can lead to schema compatibility issues if a schema registry is used.

Additionally, as per my explorations, it seems that connectors with Debezium event flattening do not face this issue.

Can anyone guide me to a solution for this?

Solution

  • I couldn't get a proper answer to this anywhere. Posting the solution that I finally ended up with.

    So it seems like there is no officially supported way of fixing this. I reached out to the Debezium community and two possible solutions that they proposed were -

    • Degrade the compatibility guarantees of schema registry to none and upgrade it to backward later sometime when all the schemas are upgraded to latest version.
    • Remove all the old schemas from the schema registry. Thus eliminating the compatibility issue.

    Implementing any of the two solutions was not possible in my case!

    For the first suggestion, it is hard to say when we will be able to upgrade the compatibility guarantees. It can only be done when all the connectors have processed at least one message after the upgrade. This is something that was very hard to assess in my case.

    For the second suggestion, removing the schema from the schema registry would cause the readers to fail to read existing messages! This is because readers query the schema registry to get the schema.

    I finally took the following approach:

    1. Paused all the connectors after the upgrade.
    2. Removed all the old schemas.
    3. Reset the offsets of all the schemas. Thus forcing them to re-process all the messages.
    4. Resumed the connectors.

    The above process ensured that all the old schemas are replaced with the newer schema.

    Although there is one catch, in my case all the downstream consumers were idempotent (safe around handling duplicate messages). This is why this approach works. If your downstream consumers are not idempotent, this approach can get your system in an awful state.