Search code examples
apache-kafkaapache-kafka-connectdebeziumchange-data-capture

Make A custom transform for kafka cdc and Debezium

I'm trying to create a custom transform for Kafka CDC.I want running a replication between MySQL and MongoDB. I prefer when a row update/insert in MySQL database the records the related to that row (A hierarchy tree structure) corresponding complex JSON document(a hierarchy JSON document) update/insert in MongoDB. Can I implement custom transform for Debezium?

Solution

  • A custom transform for Debezium is no different from a custom SMT for Connect framework, ignoring the defined scehma of the Debezium event.

    1. Create a Java project with a dependency on at least org.apache.kafka: connect-transforms
    2. Implement the Transformation interface.
      • Ideally, add unit tests for this as well
      • You can look into the Kafka source code for how existing ones are implemented.
    3. Package it as a JAR. If it needs external dependencies, a shaded JAR will work as long as you relocate any packages that Kafka itself will have on its classpath. For any of those, you should ensure provided scope; for example, kafka-clients or jackson-related libraries.
    4. Copy the JAR into a directory listed on the plugin.path of the connect worker(s) configs.

    https://docs.confluent.io/platform/current/connect/transforms/custom.html