pom.xml dependencies for spark while using scala 2.12.10

Question

These Apache Spark dependencies are not working, while working with scala 2.12.10

<dependencies>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.12.10</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>3.0.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.12</artifactId>
        <version>3.0.1</version>
    </dependency>
</dependencies>

Error while running spark app from IntelliJ

Exception in thread "main" java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V at org.apache.spark.SparkConf$DeprecatedConfig.(SparkConf.scala:784) at org.apache.spark.SparkConf$.(SparkConf.scala:605) at org.apache.spark.SparkConf$.(SparkConf.scala) at org.apache.spark.SparkConf.set(SparkConf.scala:94) at org.apache.spark.SparkConf.set(SparkConf.scala:83) at org.apache.spark.SparkConf.setMaster(SparkConf.scala:115) at org.apache.spark.SparkContext$.updatedConf(SparkContext.scala:2717) at org.apache.spark.SparkContext.(SparkContext.scala:153)

However, this set of dependencies work perfectly fine with the same spark app.

    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.11.8</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.4.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.4.1</version>
    </dependency>

Code snippet -

import org.apache.spark.SparkContext
import org.apache.log4j.{Level, Logger}
object Testing1
    {
      def main(args : Array[String]): Unit = {
        Logger.getLogger("org").setLevel(Level.OFF)
        val sc = new SparkContext("local[*]" , "SparkDemo")
        val lines = sc.textFile("sample.txt");
        val words = lines.flatMap(line => line.split(' '))
        val wordsKVRdd = words.map(x => (x,1))
        val count = wordsKVRdd.reduceByKey((x,y) => x + y).map(x => (x._2,x._1)).sortByKey(false).map(x => (x._2, x._1)).take(10)
        count.foreach(println)
      }
    }

Answer 1

This errors says about Scala version incompatibility. You either have another dependency that depends on the Scala 2.11, or you just need to do mvn clean to get rid of the old classes compiled with Scala 2.11. Also check the version of Scala configured in the Project's settings.