Search code examples
scalaapache-sparkintellij-ideaapache-spark-sqlimplicit

Overloaded method value json with alternatives: (jsonRDD: org.apache.spark.rdd.RDD[String]) using Spark in IntelliJ

I am trying to convert a JSON String jsonStr into a Spark Dataframe in Scala. Using InteliJ for this purpose.

val spark = SparkSession.builder().appName("SparkExample").master("local[*]").getOrCreate()
val sc = spark.sparkContext
import spark.implicits._
var df = spark.read.json(Seq(jsonStr).toDS)
df.show()

Getting the following error while compilation/ building the project using Maven.

Error:(243, 29) overloaded method value json with alternatives: (jsonRDD: org.apache.spark.rdd.RDD[String])org.apache.spark.sql.DataFrame (jsonRDD: org.apache.spark.api.java.JavaRDD[String])org.apache.spark.sql.DataFrame (paths: String*)org.apache.spark.sql.DataFrame (path: String)org.apache.spark.sql.DataFrame cannot be applied to (org.apache.spark.sql.Dataset[String]) var df = spark.read.json(Seq(jsonStr).toDS)

Note: I was not getting the error while building with SBT.

Solution

  • Below method was introduced in Spark 2.2.0

    def json(jsonDataset: Dataset[String]): DataFrame =

    Please correct your version of Spark in maven's pom.xml file