Search code examples
mysqlapache-kafkaapache-kafka-connectdebezium

Debezium kafka connect connector not updating successfully

I am trying to CDC using Debezium and Kafka. I have registered a connector with 3 tables in table.whitelist property and it is working properly. Here is the configuration of running connector.

{"name":"xoom-eds-extactor","config":{"connector.class":"io.debezium.connector.mysql.MySqlConnector","database.user":"debezium","database.server.id":"1","database.hostname":"qa514dtc001.ord.qa001.xoom.com","database.password":"xoom123","database.history.kafka.bootstrap.servers":"eds-kafka1:9092","database.history.kafka.topic":"dbhistory.inventory","name":"xoom-eds-extactor","database.server.name":"MySQL-Database-Docker","database.port":"3306","include.schema.changes":"true","table.whitelist":"xoom_eds_extraction_src_db_mock.postal_code_mock,xoom_eds_extraction_src_db_mock.account_number_mock,xoom_eds_extraction_src_db_mock.credit_card_number_mock,xoom_eds_extraction_src_db_mock.ip_address_details_mock,xoom_eds_extraction_src_db_mock.university_details_mock"},"tasks":[{"connector":"xoom-eds-extactor","task":0}],"type":"source"}

Now I want to update the connector and add two more tables from the same database. I am using the following command and config file to update the connector.

Command : 

curl -X PUT -H "Content-Type: application/json" --data @xoom-eds-extactor-conf.json "http://eds-kafka1:8083/connectors/xoom-eds-extactor/config"

Config json : 

{
   "name":"xoom-eds-extactor",
   "connector.class":"io.debezium.connector.mysql.MySqlConnector",
   "database.hostname":"qa514dtc001.ord.qa001.xoom.com",
   "database.port":"3306",
   "database.user":"debezium",
   "database.password":"xoom123",
   "database.server.id":"1",
   "database.server.name":"MySQL-Database-Docker",
   "database.history.kafka.bootstrap.servers":"eds-kafka1:9092",
   "database.history.kafka.topic":"dbhistory.inventory",
   "include.schema.changes":"true",
"table.whitelist":"xoom_eds_extraction_src_db_mock.postal_code_mock,xoom_eds_extraction_src_db_mock.account_number_mock,xoom_eds_extraction_src_db_mock.credit_card_number_mock,xoom_eds_extraction_src_db_mock.ip_address_details_mock,xoom_eds_extraction_src_db_mock.university_details_mock"
}

This executes successfully but when I list the Kafka topics only the last 3 topics are present 2 new topics are not added.

MySQL-Database-Docker
MySQL-Database-Docker.xoom_eds_extraction_src_db_mock.account_number_mock
MySQL-Database-Docker.xoom_eds_extraction_src_db_mock.credit_card_number_mock
MySQL-Database-Docker.xoom_eds_extraction_src_db_mock.postal_code_mock
__consumer_offsets
connect-configs
connect-offsets
connect-status
dbhistory.inventory

Can anyone out there help me with this? Thanks in advance.

Solution

  • As far as I know, there was an old pending PR related to this issue and it should have been fixed in 0.9.1 release. But if you are using some old version you can simply follow these steps.

    • Add the new tables, but don't write to them yet
    • Shut down the connector
    • Change the filter configuration and set snapshot mode to "schema_only_recovery"
    • Delete (or rename) the existing DB history topic
    • Restart the connector; this will re-create the internal history of the DB schema and then continue streaming from the previously recorded offset
    • Begin writing to the new tables

    At least this is what we do(we are still using 0.8.2).

    Source : debezium-google-group.