Kafka-Connect Cassandra Sink Connector Not Pushing Data To Cassandra

Question

I have created Kafka Standalone.properties file to built a connection. The file is placed at home/kafka/config/connect-standalone.properties as given below:

bootstrap.servers=localhost:9092

key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false

offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000

plugin.path=/home/kafka-connect-cassandra-sink-1.4.0/kafka-connect-cassandra-sink-1.4.0.jar

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000


#listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://localhost:9092

At second step I have added kafka-connect-cassandra-sink-1.4.0 file. The file is placed at the path home/kafka-connect-cassandra-sink-1.4.0 , the file is given below:

name=users-sink
connector.class=com.datastax.oss.kafka.sink.CassandraSinkConnector
tasks.max=10

loadBalancing.localDc=datacenter1
contactPoints=localhost
port=9042

username=...
password=...

topics=demo
topic.demo.demo.users.mapping=lastname=value.lastname, firstname=value.firstname, email=value.email

On my PC Zookeeper is already running and I have also started the Kafka with the following command bin/kafka-server-start.sh config/connect-standalone.properties

Now for connecting kafka to sink connector I used the following command sudo kafka/config/connect-standalone.properties kafka-connect-cassandra-sink-1.4.0/conf/cassandra-sink-standalone.properties &> standalone-mode.log &

My standalone-mode.log file is empty(I am assuming that means no error because in previous attempts there were errors in this log file which i have resolved ).

Now for uploading the text file through kafka sink connector in cassandra I have used the following command cat Desktop/users.txt | kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic demo --property "parse.key=true" --property "key.separator=:"; sleep 10;

Again there is no error on terminal. But Now the problem is when I am writing a query to view the data in cassandra database the table is empty:

user.txt file is given below:

Pruitt:{"lastname":"Pruitt", "firstname":"Allie", "email":"allie@example.com"}
Krause:{"lastname":"Krause", "firstname":"Duncan", "email":"duncan@example.com"}
Chase:{"lastname":"Chase", "firstname":"Juana", "email":"juana@example.com"}
Estrada:{"lastname":"Estrada", "firstname":"Edward", "email":"edward@example.com"}
Singleton:{"lastname":"Singleton", "firstname":"Marie", "email":"Marie@example.com"}
Poole:{"lastname":"Poole", "firstname":"Olivia", "email":"olivia@example.com"}
Marks:{"lastname":"Marks", "firstname":"Timothy", "email":"timothy@example.com"}
Suarez:{"lastname":"Suarez", "firstname":"Claud", "email":"claud@example.com"}
Sloan:{"lastname":"Sloan", "firstname":"Eloy", "email":"eloy@example.com"}
Rodriguez:{"lastname":"Rodriguez", "firstname":"Gale", "email":"gale@example.com"}
Bautista:{"lastname":"Bautista", "firstname":"Constance", "email":"Constance@example.com"}
Mcintyre:{"lastname":"Mcintyre", "firstname":"Donte", "email":"donte@example.com"}
Lang:{"lastname":"Lang", "firstname":"Willa", "email":"willa@example.com"}
Richmond:{"lastname":"Richmond", "firstname":"Dionne", "email":"dionne@example.com"}

Answer 1

started the Kafka with the following command bin/kafka-server-start.sh config/connect-standalone.properties

You don't start brokers with the Connect properties, you need to use server.properties

Plus did you verify Kafka is actually running by creating topics and other tasks mentioned in the official quickstart?

standalone-mode.log file is empty(I am assuming that means no error

Not necessarily. You're not capturing stderr with that command. e.g 2>&1

now connecting kafka to sink connector I used the following command sudo kafka/config/connect-standalone.properties kafka-connect-cassandra-sink-1.4.0/conf/...

received the following error : command not found kafka/config/connect-standalone.properties...

Properties files are not executable. You need to run connect-standalone.sh which takes those two connect properties files as arguments