apache-kafka apache-kafka-connect debezium

Debezium MySQL connector error: Encountered change event for table whose schema isn't known to this connector

I've created a Debezium connector to listen binlog event from Mysql by this config:

{
  "connector.class": "io.debezium.connector.mysql.MySqlConnector",
  "topic.creation.default.partitions": "10",
  "database.history.kafka.topic": ...,
  "include.schema.changes": "false",
  "tombstones.on.delete": "false",
  "topic.creation.datalake.partitions": "10",
  "topic.prefix": ...,
  "schema.history.internal.kafka.topic": ...,
  "database.user": ...,
  "database.server.id": ...,
  "database.history.kafka.bootstrap.servers": ...,
  "database.server.name": ...,
  "schema.history.internal.kafka.bootstrap.servers": ...,
  "database.port": ...,
  "database.hostname": ...,
  "database.password": ...,
  "name": ...,
  "table.include.list": "USERS_ACTIVITY",
  "snapshot.mode": "schema_only"
}

This connector has been working fine for me.

Today I tried to update this connector's config, set include.schema.changes to true. But after I updated the config, the connector failed when next event arrive. The error message was:

org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
    at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:374)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:874)
    at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1246)
    at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1072)
    at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631)
    at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.debezium.DebeziumException: Error processing binlog event
    ... 7 more
Caused by: io.debezium.DebeziumException: Encountered change event for table htmtalent.USERS_ACTIVITY whose schema isn't known to this connector
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:650)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:691)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:628)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$13(MySqlStreamingChangeEventSource.java:845)
    at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:354)
    ... 6 more

Why is this error happening?

Solution

This error happens when binlog fails to capture a DDL. I am using Aurora MySQL and face this error when AWS patches the Database binaries and restarts. Here what I do -

Delete or rename the history topic
change the connector snapshot.mode to schema_only_recovery

Reference - https://tech.bigbasket.com/a-year-and-a-half-with-debezium-f4f323b4909d

Refer What if it fails section

This article also suggests deleting the connector config alongwith history topic.

I would suggest you to just delete or change name of the history topic first and see if you can recover from error by modifying the connector config

Otherwise you can try as per the article

Also, in some unfortunate cases both the above approaches don't work. Then I have to create a new connector with snapshot.select.statement.overrides and snapshot.mode: initial