Spark : java.lang.NoClassDefFoundError: scala/collection/mutable/ArraySeq$ofRef

Question

I am trying to run a simple word count program with spark-submit and getting an exception.

Exception in thread "main" java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/collection/mutable/ArraySeq$ofRef at SparkWordCount$.main(SparkWordCount.scala:18)

The code, starting with line 18 is

val count = input.flatMap(line ⇒ line.split(" "))
    .map(word ⇒ (word, 1))
    .reduceByKey(_ + _)

My environment:

• Windows 10
• java version "1.8.0_221"
• spark-shell shows : Spark version 2.4.4 (Using Scala version 2.11.12)
• scala -version command shows Scala code runner version 2.13.1

Answer 1

As stated in the comments, the solution is to use for development the same version of Scala that you will use on the cluster.