Search code examples
scalaapache-spark

What is imported with spark.implicits._?

What is imported with import spark.implicits._? Does "implicits" refer to some package? If so, why could I not find it in the Scala Api documentation on https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.package?

Solution

  • Scala allows you to import "dynamically" things into scope. You can also do something like that:

    final case class Greeting(hi: String)

def greet(greeting: Greeting): Unit = {
  import greeting._ // everything in greeting is now available in scope
  println(hi)
}

    The SparkSession instance carries along some implicits that you import in your scope with that import statement. The most important thing that you get are the Encoders necessary for a lot of operations on DataFrames and Datasets. It also brings into the scope the StringContext necessary for you to use the $"column_name" notation.

    The implicits member is an instance of SQLImplicits, whose source code (for version 2.3.1) you can view here.