How to create a sparkSession using builder in Java ? I have tried this :
sparkSession = SparkSession.builder()
.master("local[*]")
.config(conf)
.appName(appName)
.getOrCreate();
But I'm getting exception in "getOrCreate". Is there any way to create a spark session using JavaSparkContext/ SparkContext ?
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.security.UserGroupInformation.getCurrentUser()Lorg/apache/hadoop/security/UserGroupInformation; at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2373) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2373) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2373) at org.apache.spark.SparkContext.(SparkContext.scala:295) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
You can create spark session object in Java as
import org.apache.spark.sql.SparkSession;
// Inside class
SparkSession spark = SparkSession
.builder()
.appName("Application Name")
.config("some-config", "some-value")
.getOrCreate();
This should work. If this didn't work can you paste whole code so that we can figure out what is going wrong.