I'm practicing running Debezium following the Official Documentation using Docker Desktop.
The commands are pretty much just copy and paste from the page except that the output registry and plugin URL for custom Kafka Connect image is replaced.
After the step creating a KafkaConnector, I didn't find the connector running at all.
I tried checking with commands below:
kubectl describe kafkaconnector debezium-connector-mysql -n debezium-example
kubectl describe kafkaconnect debezium-connect-cluster -n debezium-example
Outputs are shown as follows: Kafka Connector:
Name: debezium-connector-mysql
Namespace: debezium-example
Labels: strimzi.io/cluster=debezium-connect-cluster
Annotations: <none>
API Version: kafka.strimzi.io/v1beta2
Kind: KafkaConnector
Metadata:
Creation Timestamp: 2023-03-24T07:46:09Z
Generation: 1
Managed Fields:
API Version: kafka.strimzi.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:strimzi.io/cluster:
f:spec:
.:
f:class:
f:config:
.:
f:database.hostname:
f:database.include.list:
f:database.password:
f:database.port:
f:database.server.id:
f:database.user:
f:schema.history.internal.kafka.bootstrap.servers:
f:schema.history.internal.kafka.topic:
f:tasks.max:
f:topic.prefix:
f:tasksMax:
Manager: kubectl-create
Operation: Update
Time: 2023-03-24T07:46:09Z
Resource Version: 206115
UID: 2aef7f90-6c5c-41d5-a664-ae3be69717a2
Spec:
Class: io.debezium.connector.mysql.MySqlConnector
Config:
database.hostname: mysql
database.include.list: inventory
database.password: <nil>
database.port: 3306
database.server.id: 184054
database.user: <nil>
schema.history.internal.kafka.bootstrap.servers: debezium-cluster-kafka-bootstrap:9092
schema.history.internal.kafka.topic: schema-changes.inventory
tasks.max: 1
topic.prefix: mysql
Tasks Max: 1
Events: <none>
Kafka Connect:
Name: debezium-connect-cluster
Namespace: debezium-example
Labels: <none>
Annotations: strimzi.io/use-connector-resources: true
API Version: kafka.strimzi.io/v1beta2
Kind: KafkaConnect
Metadata:
Creation Timestamp: 2023-03-24T07:33:07Z
Generation: 1
Managed Fields:
API Version: kafka.strimzi.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:strimzi.io/use-connector-resources:
f:spec:
.:
f:bootstrapServers:
f:build:
.:
f:output:
.:
f:image:
f:type:
f:plugins:
f:config:
.:
f:config.providers:
f:config.providers.secrets.class:
f:config.storage.replication.factor:
f:config.storage.topic:
f:group.id:
f:offset.storage.replication.factor:
f:offset.storage.topic:
f:status.storage.replication.factor:
f:status.storage.topic:
f:topic.creation.enable:
f:replicas:
f:version:
Manager: kubectl-create
Operation: Update
Time: 2023-03-24T07:33:07Z
API Version: kafka.strimzi.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
f:connectorPlugins:
f:labelSelector:
f:observedGeneration:
f:replicas:
f:url:
Manager: strimzi-cluster-operator
Operation: Update
Subresource: status
Time: 2023-03-24T07:42:02Z
Resource Version: 201434
UID: 145f9996-73f9-4517-8b10-146081cf635a
Spec:
Bootstrap Servers: debezium-cluster-kafka-bootstrap:9092
Build:
Output:
Image: [REDACTED]
Type: docker
Plugins:
Artifacts:
Type: tgz
URL: https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/2.1.3.Final/debezium-connector-mysql-2.1.3.Final-plugin.tar.gz
Name: debezium-mysql-connector
Config:
config.providers: secrets
config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider
config.storage.replication.factor: -1
config.storage.topic: connect-cluster-configs
group.id: connect-cluster
offset.storage.replication.factor: -1
offset.storage.topic: connect-cluster-offsets
status.storage.replication.factor: -1
status.storage.topic: connect-cluster-status
topic.creation.enable: true
Replicas: 1
Version: 3.4.0
Status:
Conditions:
Last Transition Time: 2023-03-24T07:42:01.787955422Z
Message: database.password: A null value is not allowed for this key
Reason: InvalidConfigParameterException
Status: True
Type: NotReady
Connector Plugins:
Class: io.debezium.connector.mysql.MySqlConnector
Type: source
Version: 2.1.3.Final
Class: org.apache.kafka.connect.mirror.MirrorCheckpointConnector
Type: source
Version: 3.4.0
Class: org.apache.kafka.connect.mirror.MirrorHeartbeatConnector
Type: source
Version: 3.4.0
Class: org.apache.kafka.connect.mirror.MirrorSourceConnector
Type: source
Version: 3.4.0
Label Selector: strimzi.io/cluster=debezium-connect-cluster,strimzi.io/name=debezium-connect-cluster-connect,strimzi.io/kind=KafkaConnect
Observed Generation: 1
Replicas: 1
URL: http://debezium-connect-cluster-connect-api.debezium-example.svc:8083
...
Somehow the connector cannot read database username and password from debezium-secret, but I don't see what could be missing.
I've spent some time googling and found this, but the error message is different:
I managed to solve the problem thanks to Jakub's comment.
The problem was due to ${...} strings were not escaped and thus replaced by the shell before passing to Kubernetes. Storing the YAML part in a file followed by using kubectl apply -f ... did the trick.