Search code examples
apache-sparkpysparkcassandraspark-cassandra-connector

Writing data to Cassandra table from Kafka topic failing

I have written simple job which consumes the data from Kafka topic and writes it to cassandra table . I can see data being printed to Console however when it comes to writing it to Cassandra job fails with -

java.lang.NoSuchMethodError: scala.collection.JavaConverters$.mapAsScalaMapConverter

I am using Kafka 3.2.1 , Cassandra 3.11.13 ,pyspark 3.3.0.

my spark-submit command -

spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.13:3.3.0,com.datastax.spark:spark-cassandra-connector_2.12:3.2.0,org.apache.kafka:kafka-clients:3.2.1 --jars /home/hdoop/jars/jsr166e-1.1.0.jar iwd_dproc_new.py
Solution

  • You are using connectors compiled for different Scala versions:

    • Kafka connector is compiled with Scala 2.13
    • Cassandra connector is compiled with Scala 2.12

    It's not clear what version of Scala is used for Spark, but you need to take version that uses Scala 2.12, because there is no Spark Cassandra Connector for Scala 2.13. And adjust Kafka connector version as well