Search code examples
postgresqlapache-kafkadebeziumdebezium-enginedebezium-server

Debezium Postgres Connector "After applying the include/exclude list filters, no changes will be captured"


I am using Debezium (a Kafka Connector) to capture Postgres database changes and I am getting an error from Debezium. Does anyone know what the error below means and perhaps offer a suggestion to fix it.

A bit more debugging info:

  • I tried both "schema.include.list": "banking" and "database.include.list": "banking"... neither works
  • I tried debezium/connect:1.4 and it works... but not with debezium/connect:1.5+ (1.9 is as high a version as is available and it does not work (same error as below)
Postgres|dbserver1|snapshot  After applying the include/exclude list filters, no changes will be captured. Please check your configuration!   [io.debezium.relational.RelationalDatabaseSchema]

I have verified (in the logs) that Kafka (and schema registry etc) is running properly, and the Debezium connector seems to have started, and Postgres iw working properly and the database and tables are created.

Below is the Debezium connector configuration:

{
    "name": "banking-postgres-connector",
    "config": {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "database.hostname": "postgres",
        "database.port": "5432",
        "database.user": "postgres",
        "database.password": "postgres",
        "database.dbname" : "banking",
        "database.server.name": "dbserver1",
        "database.include.list": "banking",
        "tasks.max": "1",
        "table.include.list": "public.x_account,public.x_party,public.x_product,public.x_transaction"
    }
}

Solution

  • After many hours debugging (plus some advice from @OneCricketeer which pointed me in the right direction), I managed to get a configuration that works with debezium/connect:1.9. The solution was to use defaults by eliminating configuration items:

    database.include.list
    schema.include.list
    

    This final working Debezium configuration is as follows:

    {
        "name": "banking-postgres-connector",
        "config": {
            "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
            "database.hostname": "postgres",
            "database.port": "5432",
            "database.user": "postgres",
            "database.password": "postgres",
            "database.dbname" : "banking",
            "database.server.name": "dbserver1",
            "tasks.max": "1"
        }
    }
    

    This does point to a minor gap in the Debezium documentation and code:

    1. The documentation should provide valid values (examples) for either "schema.include.list" or "database.include.list" since adding the database name alone for either value does not seem to work for postgres...
    2. It would be great to get more information from the logs... the warning message was both hard to find/understand and left one with little recourse. Since this is a situation where no data is captured, it may warrant a higher importance (log an error?)

    NOTE: I offer the above as humble suggestions because I find Debezium to be an OUTSTANDING product!