Search code examples
scalaapache-sparkapache-spark-sqlapache-zeppelin

value registerAsTable is not a member of org.apache.spark.sql.DataFrame


i am running below code in Zeppelin 0.7

%spark 
//val sc: SparkContext // An existing SparkContext.
sc
import sqlContext.implicits._
import org.apache.spark.sql._
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.SQLContext;
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val people = sqlContext.jsonFile("/Users/asdf/Desktop/people.json")
people.printSchema()
people.show()
people.select("name").show()
people.toDF().registerAsTable("people")

Its working till people.select("name").show() but throws error at last line, below is the error:

 +-------+
 |   name|
 +-------+
 |Michael|
 |   Andy|
 | Justin|
 +-------+

<console>:230: error: value registerAsTable is not a member of org.apache.spark.sql.DataFrame
          people.toDF().registerAsTable("people")

asper my knowledge i imported all required and converted it to df before registering it as table. what am i missing here?


Solution

  • Below code worked for me, the issue was with Zeppelin that we should not create a SQLContext Zeppelin provides its own.

    %spark 
    import sqlContext.implicits._
    import org.apache.spark.sql._
    import org.apache.spark.sql.DataFrame;
    import org.apache.spark.sql.SQLContext;
    
    val people = sqlContext.jsonFile("/Users/asdf/Desktop/people.json")
    people.printSchema()
    people.show()
    people.select("name").show()
    people.toDF().registerTempTable("people")
    

    Then above created Temp table can be used to Query for interactive charts in %sql notebook of Zeppelin