Task state of a connector is getting failed with error:
org.apache.kafka.connect.errors.ConnectException: java.lang.NoClassDefFoundError
I am running kafka connect cluster in distributed mode and I am using kafka(0.10.0.2.5) connect of ambari deployment.
I gave debezium mysql connector path using export CLASSPATH=/path to connector/.
Loaded connector configuration into Kafka Connect using the following command:
curl -i -X POST -H "Accept:application/json" \
-H "Content-Type:application/json" http://localhost:8083/connectors/ \
-d '{
"name": "MYSQL_CONNECTOR",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"database.hostname": "10.224.21.36",
"database.port": "3306",
"database.user": "root",
"database.password": "shobhna",
"database.server.id": "1",
"database.server.name": "demo",
"database.history.kafka.bootstrap.servers": "slnxhadoop04.noid.in:6669",
"database.history.kafka.topic": "dbhistory.demo" ,
"include.schema.changes": "true"
}
}'
Now after checking connector status, I am getting error:
- {"name":"MYSQL_CONNECTOR","connector":{"state":"RUNNING","worker_id":"172.26.177.115:8083"},
"tasks":[{"state":"FAILED","trace":"org.apache.kafka.connect.errors.ConnectException:
java.lang.NoClassDefFoundError:
org/apache/kafka/clients/admin/AdminClient\n\tat
io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:218)\n\tat
io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:45)\n\tat
org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:137)\n\tat
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)\n\tat
org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)\n\tat
java.util.concurrent.Executors$RunnableAdapter.cal(Executors.java:511)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat
java.lang.Thread.run(Thread.java:745)\nCaused by:
java.lang.NoClassDefFoundError:
org/apache/kafka/clients/admin/AdminClient\n\tat
io.debezium.relational.history.KafkaDatabaseHistory.initializeStorage(KafkaDatabaseHistory.java:336)\n\tat
io.debezium.connector.mysql.MySqlSchema.intializeHistoryStorage(MySqlSchema.java:260)\n\tat
io.debezium.connector.mysql.MySqlTaskContext.initializeHistoryStorage(MySqlTaskContext.java:194)\n\tat
io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:126)\n\t...
9 more\nCaused by: java.lang.ClassNotFoundException:
org.apache.kafka.clients.admin.AdminClient \n\tat
java.net.URLClassLoader.findClass(URLClassLoader.java:381)\n\tat
java.lang.ClassLoader.loadClass(ClassLoader.java:424)\n\tat
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)\n\tat
java.lang.ClassLoader.loadClass(ClassLoader.java:357)\n\t
It can't find a builtin Kafka class, not your Connector
NoClassDefFoundError: org/apache/kafka/clients/admin/AdminClient
...
i am using kafka(0.10.0.2.5)
Make sure you're running 1) a Connect Server version that matches your Kafka broker 2) using a Connector that uses code for that version of Connect
For example, AdminClient only exists in Kafka 0.11+.
In the recent HDP releases, you get Kafka 1.1 (different than 0.11), and this is the version that the latest Debezium is built and tested against https://debezium.io/docs/releases/
Debezium needs the AdminClient to make and register topic information, so I'm not sure if it'll work on old version such as 0.10
Its stated in the Kafka wiki that newer versions of Connect Server can communicate with old brokers, but the protocol used by the Connector classes is up for debate.