Search code examples
javaserializationapache-sparkapache-spark-mllibcollaborative-filtering

How to serialize apache spark's MatrixFactorizationModel in Java

I am building a recommendation system using Apache Spark MLlib and Java. Once the MatrixFactorizationModel is built, I have serialized it as a java object and when retrieving the model, I am getting the following exception.

Caused by: java.lang.ClassNotFoundException: org.apache.spark.OneToOneDependency cannot be found by org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a39653b

Any idea how to fix this?

Solution

  • You are usiong incompatible versions of scala and spark. Spark 1.4.0 need scala version 2.11.6 but are using scala version 2.10.4(from your error stack).

    PS:You could find out compatible versions from here