I have been connecting to Hive (which is in cluster) from my local system with spark in scala IDE , finally placed my hive-site.xml properly in Spark/conf folder and in class path and able to connect to metastore but unable to access hive tables.
How to change the user for HiveContext
15/12/22 10:28:42 INFO ParseDriver: Parsing command: show tables 15/12/22 10:28:43 INFO ParseDriver: Parse Completed 15/12/22 10:28:45 INFO metastore: Trying to connect to metastore with URI thrift://Server.com:9083 15/12/22 10:28:46 INFO JniBasedUnixGroupsMapping: Error getting groups for XXXX: The user name could not be found.
15/12/22 10:28:46 WARN UserGroupInformation: No groups available for user XXXXX 15/12/22 10:28:46 INFO metastore: Connected to metastore.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/tez/dag/api/SessionNotRunning at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:353) at org.apache.spark.sql.hive.HiveContext$$anonfun$4.apply(HiveContext.scala:235) at org.apache.spark.sql.hive.HiveContext$$anonfun$4.apply(HiveContext.scala:231) at scala.Option.orElse(Option.scala:257) at org.apache.spark.sql.hive.HiveContext.x$3$lzycompute(HiveContext.scala:231) at org.apache.spark.sql.hive.HiveContext.x$3(HiveContext.scala:229) at org.apache.spark.sql.hive.HiveContext.hiveconf$lzycompute(HiveContext.scala:229) at org.apache.spark.sql.hive.HiveContext.hiveconf(HiveContext.scala:229) at org.apache.spark.sql.hive.HiveMetastoreCatalog.(HiveMetastoreCatalog.scala:55) at org.apache.spark.sql.hive.HiveContext$$anon$1.(HiveContext.scala:253) at org.apache.spark.sql.hive.HiveContext.catalog$lzycompute(HiveContext.scala:253) at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:253) at org.apache.spark.sql.hive.HiveContext$$anon$3.(HiveContext.scala:263) at org.apache.spark.sql.hive.HiveContext.analyzer$lzycompute(HiveContext.scala:263) at org.apache.spark.sql.hive.HiveContext.analyzer(HiveContext.scala:262) at org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQLContext.scala:411) at org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.scala:411) at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58) at org.apache.spark.sql.SchemaRDD.(SchemaRDD.scala:108) at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:102) at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:106) at com.Spark.ConnectToHadoop.CountWords$.main(CountWords.scala:50) at com.Spark.ConnectToHadoop.CountWords.main(CountWords.scala) Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.api.SessionNotRunning at java.net.URLClassLoader$1.run(Unknown Source) at java.net.URLClassLoader$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) ... 23 more
This answer that has nothing to do with your question, but then, your question has nothing to do with the Exception shown...
In the hive-site.xml that you provided to Spark, change propertyhive.execution.engine from "tez" (requires TEZ run-time) to default "mr".