Search code examples
scalaapache-sparkapache-spark-sqlparquet

Scala Error: value registerTempTable is not a member of org.apache.spark.sql.SchemaRDD

I want to read a Parquet file. I write this code in spark-shell:

val logFile = "hdfs://byd0085/"
val sqlContext=new org.apache.spark.sql.SQLContext(sc)
val parquetFile = sqlContext.parquetFile(logFile)
parquetFile.registerTempTable("parquetFile")

By calling parquetFile.registerTempTable("parquetFile"), there is a error error: value registerTempTable is not a member of org.apache.spark.sql.SchemaRDD . But parquetFile is indeed a SchemaRDD and registerTempTable is indeed a function of org.apache.spark.sql.SchemaRDD. I'm new to Scala and Spark and I can't figure out the reason. Anyone can help?

Spark Version 1.0.0 Scala Version 2.10.4

Solution

  • In Spark 1.0, this was called registerAsTable