How to change spark session in spark shell?

Question

I have been experimenting with multiple spark sessions using the same spark context. Once the spark-shell is launched, I can create a new spark session using the following.

val spark2 = spark.newSession();
spark2: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@43e869ea

scala> spark
res0: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@1390db9e

scala> spark2
res1: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@43e869ea

However, even though the session is created and available, it is still not an active session. Each spark session has its own temporary views. If I am creating a data frame, it is already associated with the default spark session.

scala> val s = Seq(1,2,3).toDF("num")
s: org.apache.spark.sql.DataFrame = [num: int]

scala> val s2 = Seq(4,5,6).toDF("num")
s2: org.apache.spark.sql.DataFrame = [num: int]

scala> s
res2: org.apache.spark.sql.DataFrame = [num: int]

scala> s.sparkSession
res3: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@1390db9e

Is there a way to switch the active spark session, so that the next operations are done using that?

Answer 1

You're probably looking for the setActiveSession(session: SparkSession): Unit method defined in companion object.

scala> val spark2 = spark.newSession();
spark2: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@43e869ea

scala> import org.apache.spark.sql.SparkSession

scala> SparkSession.setActiveSession(spark2)

scala> SparkSession.active
res1: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@43e869ea