Spark running Liblinear unable to load JBLAS jar

I'm running spark 1.4.0, hadoop 2.7.0, and JDK 7. I'm trying to run the example code of Liblinear presented here.

The liblinear jar works, however when training the model it can't find the JBLAS library. I've tried including a JBLAS library in the --jars option when launching spark, as well as installing the jar with maven (although I must add I am a newbie to spark as well as maven so I probably did it wrong).

The specific error thrown is this:

    java.lang.NoClassDefFoundError: org/jblas/DoubleMatrix
    at tw.edu.ntu.csie.liblinear.Tron.tron(Tron.scala:323)
    at tw.edu.ntu.csie.liblinear.SparkLiblinear$.tw$edu$ntu$csie$liblinear$SparkLiblinear$$train_one(SparkLiblinear.scala:32)`

when running this line:

    val model = SparkLiblinear.train(data, "-s 0 -c 1.0 -e 1e-2")`

Thanks.

Solution

java.lang.NoClassDefFoundError: org/jblas/DoubleMatrix

It seems that you did not add jblas jar. The solution could be:

$ export SPARK_CLASSPATH=$SPARK_CLASSPATH:/path/to/jblas-1.2.3.jar

After that, it would work fine.

Hope this helps,

Le Quoc Do