Search code examples
postgresqlapache-kafkaapache-kafka-connectdebeziumstrimzi

milliseconds timestamp in debezium postgres connector


the segments are rolling over so quickly in our Kafka cluster, and it might be because we are producing records into the kafka cluster with a micro second timestamp, and Kafka expects it to be in milliseconds. Is it possible to change it to milliseconds in debezium postgres connector? Which config should I use to make data coming in milliseconds? Our kafka cluster is being unstable due to this issue.

config

   plugin.name: wal2json_streaming
   max.queue.size: 65536
   max.batch.size: 16384
   slot.max.retries: 20
   slot.name: {{ name }}
   database.hostname: {{ hostname }}
   database.port: 5432
   database.user: {{ user }}
   database.password: {{ password }}
   database.dbname: {{ name }}
   schema.whitelist: public
   table.whitelist:  public.outbox
   database.server.name: {{ server.name }}
   database.server.id: {{ server.id }}
   database.history.kafka.bootstrap.servers: {{ bootstrapServers }}
   

Solution

  • This issue is fixed in debezium 1.1.0 version