Search code examples
javaapache-sparkapache-spark-sqlspark-avro

Spark 1.6 load specific partition in dataframe keeping partition field

We have an avro with partitioned like this:

table
--a=01
--a=02

We want to load the data from a single partition keeping the partition column a. I found this stackoverflow question and I applied the suggested snippet:

DataFrame df = hiveContext.read().format(format).
option("basePath", "path/to/table/").
load("path/to/table/a=01")

but when I try to read the field a it says: 

org.apache.spark.sql.AnalysisException: cannot resolve 'a' given input columns: [b, c, d]

Is it possible in Spark Java API 1.6?

Solution

  • Simply, the answer is... no, it's not possible.