Search code examples
scalaapache-sparksbtspark-submit

Runtime error on Scala Spark 2.0 code

I have the following code:

import org.apache.spark.sql.SparkSession
        .
        .
        .
    val spark = SparkSession
      .builder()
      .appName("PTAMachineLearner")
      .getOrCreate()

When it executes, I get the following error:

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
    at org.apache.spark.sql.SparkSession$Builder.config(SparkSession.scala:750)
    at org.apache.spark.sql.SparkSession$Builder.appName(SparkSession.scala:741)
    at com.acme.pta.accuracy.ml.PTAMachineLearnerModel.getDF(PTAMachineLearnerModel.scala:52)

The code compiles and builds just fine. Here are the dependencies:

scalaVersion := "2.11.11"
libraryDependencies ++= Seq(
  // Spark dependencies
  "org.apache.spark" %% "spark-hive" % "2.1.1",
  "org.apache.spark" %% "spark-mllib" % "2.1.1",
  // Third-party libraries
  "net.sf.jopt-simple" % "jopt-simple" % "5.0.3",
  "com.amazonaws" % "aws-java-sdk" % "1.3.11",
  "org.apache.logging.log4j" % "log4j-api" % "2.8.2",
  "org.apache.logging.log4j" % "log4j-core" % "2.8.2",
  "org.apache.logging.log4j" %% "log4j-api-scala" % "2.8.2",
  "com.typesafe.play" %% "play-ahc-ws-standalone" % "1.0.0-M9",
  "net.liftweb" % "lift-json_2.11" % "3.0.1"
)

I am executing the code like this:

/Users/paulreiners/spark-2.1.1-bin-hadoop2.7/bin/spark-submit \
      --class "com.acme.pta.accuracy.ml.CreateRandomForestRegressionModel" \
      --master local[4] \
      target/scala-2.11/acme-pta-accuracy-ocean.jar \

I had this all running with Spark 1.6. I'm trying to upgrade to Spark 2, but am missing something.

Solution

  • The class ArrowAssoc is indeed present in your Scala library. See this Scala doc . But you are getting error in Spark library. So obviously, Spark version you are using is not compatible with Scala ver 2.11 as it is probably compiled with older Scala version. If you see this older Scala API doc , the ArrowSpec has changed a lot. e.g. it is implicit now with lots of implicit dependencies. Make sure your Spark and Scala version are compatible.