Search code examples
postgresqlapache-kafkadebezium

Kakfa topic not recreated by Debezium after deleting


Dezebium is not recreating topics for a table after they are deleted. In this question I was able to create a Kafka topic to stream updates from the table specified in table.include.list. I deleted the topic and reran bin/connect-standalone.sh with the same config, reproduced below, but the same topic is not recreated. I removed table.include.list, reran bin/connect-standalone.sh, and all the topics other than for table myschema.ipaddrs were recreated. I deleted those topics, reran bin/connect-standalone.sh, and no topics are recreated.

name=testdb
connector.class=io.debezium.connector.postgresql.PostgresConnector
topic.prefix=test
database.hostname=localhost
database.port=5432
database.user=postgres
database.password=root
database.dbname=testdb
database.server.name=testdb
table.include.list=myschema.ipaddrs
plugin.name=pgoutput

Before running bin/connect-standalone.sh, I ran curl -i -X DELETE localhost:8083/connectors/testdb/ to delete the connector. I don't have any other connectors.

I see these logs related to the above table. I don't see any errors.

[2022-10-20 11:52:42,655] INFO [testdb|task-0]    table.include.list = myschema.ipaddrs (io.debezium.connector.common.BaseSourceTask:131)
[2022-10-20 11:52:42,894] INFO [testdb|task-0] REPLICA IDENTITY for 'myschema.ipaddrs' is 'FULL'; UPDATE AND DELETE events will contain the previous values of all the columns (io.debezium.connector.postgresql.PostgresSchema:100)
[2022-10-20 11:52:42,965] INFO [testdb|task-0] REPLICA IDENTITY for 'myschema.ipaddrs' is 'FULL'; UPDATE AND DELETE events will contain the previous values of all the columns (io.debezium.connector.postgresql.PostgresSchema:100)

The topics are not recreated if I change either the name of the connector or topic.prefix

I observe this same behavior if I run bin/connect-distributed.sh and pass the above config as JSON to the API.


Solution

  • The topics are not created if the underlying table has no data. The topic was created once I added a row into the table.