Debezium Kafka Connect requires restart for PostgreSQL cdc to work

Question

I'm trying to do cdc with debezium kafka connect with the following setup:

• Zookeeper
• Kafka
• PostgreSQL
• Kafka Connect Debezium

I'm also configuring and starting them up in that order.

After starting up Kafka Connect Debezium I do a POST to configure it:

{
  "name": "postgres-boxes-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "172.17.0.4",
    "database.port": "5432",
    "database.user": "postgres",
    "database.password": "postgres",
    "database.dbname" : "postgres",
    "database.server.name": "fullfillment",
    "table.whitelist": "public.boxes"
  }
}

After starting my watcher I see the following warning/error:

WARN [Consumer clientId=consumer-1, groupId=console-consumer-53726] Error while fetching metadata with correlation id 2 : {fullfillment.public.boxes=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

Now when I restart Kafka Connect Debezium everything works fine without errors and the cdc works just fine.

I'm now trying to figure out why I have to restart Kafka Connect for it to work and hopefully it can be avoided. If more info is required I'm happy to provide more.

Answer 1

You have to restart the worker in order to pick up the connector plugin when you install it. Other than that, there is no need to restart the Connect worker for the connector to work.

It's just a WARN that you saw, which means the task continues executing. The LEADER_NOT_AVAILABLE is often seen when a new topic gets created - and then subsequently created (If you see LEADER_NOT_AVAILABLE continuously and it doesn't stop then it suggests that the topic has not been automatically created and you may need to create it manually).