Context :
Sorry to bother you with I am struggling to install a Cassandra-Spark connector. My goal is to install it to use Spark-SQL since Cassandra has strong limitations to do requests.
I have :
It matches 3.1.0 connector (https://github.com/datastax/spark-cassandra-connector).
I tried to submit a simple file named titi.py (for the example) to check if it works with that command :
spark-submit dev_spark/titi.py \
--packages com.datastax.spark:spark-cassandra-connector_2.12:3.1.0
Error part :
I get that error message :
Traceback (most recent call last): File "/home/ubuntu/dev_spark/titi.py", line 12, in \\\<module\\\> .load()
File "/home/ubuntu/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 210, \
in load File "/home/ubuntu/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1304, \
in \__call_\_ File "/home/ubuntu/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, \
in deco File "/home/ubuntu/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 326, \
in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o36.load. : \
java.lang.ClassNotFoundException: Failed to find data source: org.apache.spark.sql.cassandra. \
Please find packages at http://spark.apache.org/third-party-projects.html \
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:689) \
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:743) \
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:266) \
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:226) \
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) \
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) \
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) \
at java.lang.reflect.Method.invoke(Method.java:498) \
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) \
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) \
at py4j.Gateway.invoke(Gateway.java:282) \
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) \
at py4j.commands.CallCommand.execute(CallCommand.java:79) \
at py4j.GatewayConnection.run(GatewayConnection.java:238) \
at java.lang.Thread.run(Thread.java:750) \
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.cassandra.DefaultSource \
at java.net.URLClassLoader.findClass(URLClassLoader.java:387) \
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) \
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) \
at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$5(DataSource.scala:663) \
at scala.util.Try$.apply(Try.scala:213) \
at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$4(DataSource.scala:663) \
at scala.util.Failure.orElse(Try.scala:224)
Python file for information
For information, in the test titi.py file :
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("CassandraIntegration") \
.config("spark.cassandra.connection.host", "tp-hadoop-33") \
.config("spark.cassandra.connection.port", "9042") \
.getOrCreate()
df = spark.read \
.format("org.apache.spark.sql.cassandra") \
.options(table="request_2", keyspace="gdelt") \
.load()
spark.stop()
Thanks for your help if you have any idea and have a nice weekend !
@andrei we used your advice to solve the problem
We used that command to install it :
spark-submit --packages com.datastax.spark:spark-cassandra-connector_2.12:3.1.0, \
com.github.jnr:jnr-ffi:2.2.7,com.github.jnr:jnr-posix:3.1.4 dev_spark/titi.py
Thank you everyone !