Search code examples
apache-kafkaapache-kafka-streams

Strategies to migrate stateful stream (KTable) and materialized views in Kafka


How the materialized view should be migrated when the stored object fields are changed (types, fields number etc.)? Could you tell me please required steps or point me a documentation? Is there a good practices? If AVRO schema evaluates? Does the forward/backward compatibility matters? How this is handled by stateful stream(KTable) in Kafka Streams API? Any articles on this topic?


Solution

  • Format/compatibility doesn't completely matter, other than you need to ensure newer schemas can deserialize older data, i.e "backwards compatible". But you could also seek any consumer to the start of the new data, if that weren't possible...

    You would need to reset the application, then rebuild all output streams and tables based on the new schemas.