Search code examples
mavenapache-sparkcassandradatastaxspark-cassandra-connector

java.lang.NoClassDefFoundError: com/datastax/spark/connector/ColumnSelector while building a jar

Am using maven project with spark, and a cassandra as a database. Am able to execute the code with 0 error, and getting the result that I want, but I can't submit the project with the jar file: here is my pom.xml :

POM.xml

my commands to submit the project:

spark-submit --master "local[*]" --class com.sparkfinal.App target/sparkfinal-1.0-SNAPSHOT.jar

the error:

ERROR

Solution

  • You need either:

    1. create an assembly jar for your project that will include all dependencies (make sure you declare Spark dependencies as provided)
    2. or specify Spark Cassandra Connector as dependency when submitting the job by using --packages com.datastax.spark:spark-cassandra-connector_2.12:3.2.0 as shown in the connector's docs

    I would recommend to use first method because I see that you have other dependencies that you'll need to provide. Also, please note that you may not need to specify java driver dependency as it will be pulled by Spark Cassandra Connector automatically.