Issue with Zeppelin on Spark-Cassandra system: Classnotfoundexception

Question

I have recently started to work with zeppelin on top of a Spark-Cassandra Cluster (Master + 3 Workers) System to run simple machine learning algorithms using the MLlib library.

Here are the libraries that I loaded to zeppelin:

%dep
z.load("com.datastax.spark:spark-cassandra-connector_2.10:1.4.0-M1")
z.load("org.apache.spark:spark-core_2.10:1.4.1")
z.load("com.datastax.cassandra:cassandra-driver-core:2.1.3")
z.load("org.apache.thrift:libthrift:0.9.2")
z.load("org.apache.spark:spark-mllib_2.10:1.4.0")
z.load("cassandra-clientutil-2.1.3.jar")
z.load("joda-time-2.3.jar")

I've tried to implement a script for linear regression. But, when I run it I get the following error message :

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 192.xxx.xxx.xxx): java.lang.ClassNotFoundException: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:344)
at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:66)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
...

The problem is that the script runs without problems using spark-submit script, which confused me.

Here is some of the code that I was trying to execute:

import org.apache.spark.SparkContext

import org.apache.spark.SparkConf

import com.datastax.spark.connector._

import com.datastax.spark.connector.cql.CassandraConnector

import org.apache.spark.mllib.regression.{LinearRegressionWithSGD, LinearRegressionModel, LabeledPoint}

import org.apache.spark.rdd.RDD



    sc.stop()
    val conf = new SparkConf(true).set("spark.cassandra.connection.host", "xxx.xxx.xxx.xxx").setMaster("spark://xxx.xxx.xxx.xxx:7077").setAppName("DEMONSTRATION")

    val sc = new SparkContext(conf)

   case class Fact(numdoc:String, numl:String, year:String, creator:Double, date:Double, day:Double, user:Double, workingday:Double, total:String) 

    val data= sc.textFile("~/Input/Data.csv »)

    val parsed = data.filter(!_.isEmpty).map {row => 
        val splitted = row.split(",")
        val Array(nd, nl, yr)=splitted.slice(0,3)
        val Array(cr, dt, wd, us, wod)=splitted.slice(3,8).map(_.toDouble)
        Fact (nd, nl, yr, cr, dt, wd, us, wod, splitted(8))
    }

    val class2id = parsed.map(_.total.toDouble).distinct.collect.zipWithIndex.map{case (k,v) => (k, v.toDouble)}.toMap

    val id2class = class2id.map(_.swap)

    val parsedData = parsed.map { i => LabaledPoint(class2id(i.total.toDouble), Array(i.creator,i.date,i.day,i.workingday))

    val model: LinearRegressionModel = LinearRegressionWithSGD.train(parsedData, 3)

Thank you in advance !

Answer 1

I finally found a solution ! In fact I shouldn't stop the SparkContext in the beginning and create a new one. But, In that case, I could not get access to Cassandra in a remote machine because by default zeppelin uses the address of the machine where it is installed as the address of Cassandra host. So I installed a new Cassandra instance there and I added it to my initial cluster, and the problem was solved.