Search code examples
apache-kafkaapache-kafka-connectdebezium

How to transform all timestamp fields according to avro scheme when using Kafka Connect?

In our database we have over 20 fields which we need to transform from long to timestamp. Why there is no generic solution to transfer all this value ?

I know I can define:

"transforms":"tsFormat",
"transforms.tsFormat.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
"transforms.tsFormat.target.type": "string",
"transforms.tsFormat.field": "ts_col1",
"transforms.tsFormat.field": "ts_col2",

but this is not solution for us. When we add new timestamp to db we need to update connector too

is there some generic solution to transform all fields according to avro schema ?

We are using debezium which for all timestamp fields create something like this:

    {
      "name": "PLATNOST_DO",
      "type": {
        "type": "long",
        "connect.version": 1,
        "connect.name": "io.debezium.time.Timestamp"
      }
    },

so how to find all type with connect.name = 'io.debezium.time.Timestamp' and transform it to timestamp

Solution

  • You'd need to write your own transform to be able to dynamically iterate of the record schema, check the types, and do the conversion.

    Thus why they are called simple message transforms.

    Alternatively, maybe take a closer look at the Debezium properties to see if there is a missing setting that alters how timestamps get produced.