Search code examples
sql-serverapache-kafkaapache-kafka-connectdebezium

Debezium SQLServerConnector for Kakfa Connect not creating topics in Kafka

I'm trying to get the SQLServerConnector from Debezium working in Kafka Connect and everything seems to be running fine, except there are no topics created. Am I missing something?

It starts the task with the right configs...

Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,424] INFO Kafka version : 2.1.1-cp1 (org.apache.kafka.common.utils.AppInfoParser:109)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,424] INFO Kafka commitId : f5b753880d5460f1 (org.apache.kafka.common.utils.AppInfoParser:110)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,425] INFO Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:860)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO Starting SqlServerConnectorTask with configuration: (io.debezium.connector.common.BaseSourceTask:42)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO    connector.class = io.debezium.connector.sqlserver.SqlServerConnector (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO    database.user = cdc_test (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO    database.dbname = kafka_sql_test (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,426] INFO    task.class = io.debezium.connector.sqlserver.SqlServerConnectorTask (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.hostname = mbvmsqlsvr001-t (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.history.kafka.bootstrap.servers = horton1:9092,horton2:9092,horton3:9092,horton4:9092 (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.history.kafka.topic = dbhistory.sql2016test (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.password = ******** (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    name = sql-server-test-connector (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.server.name = mbvmsqlsvr001-t (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    database.port = 1433 (io.debezium.connector.common.BaseSourceTask:44)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,427] INFO    table.whitelist = dbo.message_table (io.debezium.connector.common.BaseSourceTask:44)

Gets the snapshot setup......

Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,430] INFO Kafka version : 2.1.1-cp1 (org.apache.kafka.common.utils.AppInfoParser:109)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,430] INFO Kafka commitId : f5b753880d5460f1 (org.apache.kafka.common.utils.AppInfoParser:110)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,522] INFO Requested thread factory for connector SqlServerConnector, id = mbvmsqlsvr001-t named = change-event-source-coordinator (io.debezium.util.Threads:23
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO Creating thread debezium-sqlserverconnector-mbvmsqlsvr001-t-change-event-source-coordinator (io.debezium.util.Threads:247)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO WorkerSourceTask{id=sql-server-test-connector-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:19
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO No previous offset has been found (io.debezium.connector.sqlserver.SqlServerSnapshotChangeEventSource:56)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO According to the connector configuration both schema and data will be snapshotted (io.debezium.connector.sqlserver.SqlServerSnapshotChangeEventSourc
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,523] INFO Snapshot step 1 - Preparing (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:85)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,524] INFO Snapshot step 2 - Determining captured tables (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:94)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,528] INFO Snapshot step 3 - Locking captured tables (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:100)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,530] INFO Schema locking was disabled in connector configuration (io.debezium.connector.sqlserver.SqlServerSnapshotChangeEventSource:84)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,530] INFO Snapshot step 4 - Determining snapshot offset (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:106)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,530] INFO Snapshot step 5 - Reading structure of captured tables (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:109)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,530] INFO Snapshot step 6 - Persisting schema history (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:113)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,531] INFO Snapshot step 7 - Snapshotting data (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:125)
Apr 16 15:30:00 horton5 connect-distributed[9469]: [2019-04-16 15:30:00,531] INFO Snapshot step 8 - Finalizing (io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource:149)

And the connector tasks are running...

# curl  http://172.24.32.10:8083/connectors/sql-server-test-connector/status | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   175  100   175    0     0  73160      0 --:--:-- --:--:-- --:--:-- 87500
{
  "name": "sql-server-test-connector",
  "connector": {
    "state": "RUNNING",
    "worker_id": "horton5:8083"
  },
  "tasks": [
    {
      "id": 0,
      "state": "RUNNING",
      "worker_id": "horton5:8083"
    }
  ],
  "type": "source"
}

The username and password have a SPID on the SQL Server and running the CDC queries to get the changes, but nothing shows up in a Kafka topic.

What am I not understanding? Am I supposed to create the topic first and then how would I config the connector to publish messages to it?

UPDATE: Here's the connector json

    curl -X POST -H "Content-Type: application/json" --data '{
  "name": "sql-server-test-connector",
  "config": {
    "connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
    "database.hostname": "mbvmsqlsvr001-t",
    "database.port": "1433",
    "database.user": "cdc_test",
    "database.password": "Password!123",
    "database.dbname": "kafka_sql_test",
    "database.server.name": "mbvmsqlsvr001-t",
    "table.whitelist": "dbo.message_table",
    "database.history.kafka.bootstrap.servers": "horton1:9092,horton2:9092,horton3:9092,horton4:9092",
    "database.history.kafka.topic": "dbhistory.sql2016test"
    }
  }' http://172.24.32.10:8083/connectors
Solution

  • FIXED.

    My own stupid mistake. I was using the debezium-connector-sqlserver-0.9.0.Alpha1-plugin.tar.gz distribution. I put the debezium-connector-sqlserver-0.9.4.Final-plugin.tar.gz in place and everything works perfect (I can see changes in the consumer nearly real time).

    I don't even know how I got a hold of the Alpha version.......