Search code examples
scalaapache-sparkapache-spark-1.6

Pivot spark scala dataframe

I am trying to use pivot method in scala-spark 

val dfOutput =  df_input.groupBy("memberlogin").pivot("country_group2").count()

However, though there isn't any compliation error while creating a jar in eclipse, while execution in spark, its giving error as --
Exception in thread "main" java.lang.NoSuchMethodError: 

org.apache.spark.sql.GroupedData.pivot(Ljava/lang/String;)Lorg/apache/spark/sql/GroupedData;

I have imported the desired libraries, 

import org.apache.spark.sql.GroupedData
import org.apache.spark.sql.functions

also, 

import sqlContext.implicits._

as few other threads claimed.

Its not working, and unable to understand, can someone help or guide in direction?

Solution

  • Pivoting feature introduced in Spark 1.6 please check this documentation Reshaping Data with Pivot in Apache Spark . Check the API Pivot Api