Search code examples
scalaapache-sparkintellij-ideasbt

Error while running sbt package: object apache is not a member of package org

When I try sbt package in my below code I get these following errors

  1. object apache is not a member of package org

  2. not found: value SparkSession

    MY Spark Version: 2.4.4
    My Scala Version: 2.11.12

My build.sbt

name := "simpleApp"

version := "1.0"

scalaVersion := "2.11.12"

//libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.4"


  libraryDependencies ++= {
   val sparkVersion = "2.4.4"
       Seq( "org.apache.spark" %% "spark-core" % sparkVersion)
  }

my Scala project

import org.apache.spark.sql.SparkSession
object demoapp {
  def main(args: Array[String]) {

val logfile = "C:/TEST/demo/hello.txt"
val spark = SparkSession.builder.appName("Simple App in Scala").getOrCreate()
val logData = spark.read.textFile(logfile).cache()
val numAs = logData.filter(line => line.contains("Washington")).count()
println(s"Lines are: $numAs")
spark.stop()
}
}
Solution

  • If you want to use Spark SQL, you also have to add the spark-sql module to the dependencies:

    // https://mvnrepository.com/artifact/org.apache.spark/spark-sql
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.4"

    Also, note that you have to reload your project in SBT after changing the build definition and import the changes in intelliJ.