No CDC generated by Kafka Debezium connector for Postgres

Question

I succeed generating CDC in a Postgres DB. Today, when I use same step to try to set up Kafka Debezium connector for another Postgres DB.

First I ran

POST http://localhost:8083/connectors

with body:

{
    "name": "postgres-kafkaconnector",
    "config": {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "tasks.max": "1",
        "database.hostname": "example.com",
        "database.port": "5432",
        "database.dbname": "my_db",
        "database.user": "xxx",
        "database.password": "xxx",
        "database.server.name": "postgres_server",
        "table.include.list": "public.products",
        "plugin.name": "pgoutput"
    }
}

which succeed without error.

Then I ran

GET http://localhost:8083/connectors/postgres-kafkaconnector/status

to check status. It returns this result without any error:

{
    "name": "postgres-kafkaconnector",
    "connector": {
        "state": "RUNNING",
        "worker_id": "10.xx.xx.xx:8083"
    },
    "tasks": [
        {
            "id": 0,
            "state": "RUNNING",
            "worker_id": "10.xx.xx.xx:8083"
        }
    ],
    "type": "source"
}

However, this time, when I updated anything in the products table. No CDC got generated.

Any idea? Any suggestion for helping further debug would be appreciate. Thanks!

Answer 1

Found the issue! It is because my Kafka Connector postgres-kafkaconnector was initially pointing to a DB (stage1), then I switched to another DB (stage2) by updating

"database.hostname": "example.com",
"database.port": "5432",
"database.dbname": "my_db",
"database.user": "xxx",
"database.password": "xxx",

However, they are using same configuration properties in the Kafka Connect I deployed in the very beginning:

config.storage.topic
offset.storage.topic
status.storage.topic

Since this connector with different DB config shared same above Kafka configuration properties, nd the database table schemas are same,

it became mess due to sharing same Kafka offset.

One simple way to fix is when deploying Kafka connector to test on different DBs, using different names such as postgres-kafkaconnector-stage1 and postgres-kafkaconnector-stage2 to avoid Kafka topic offset mess.