I am trying to run Mobius HiveDataFrame Example but not able to run.
var jsonFilePath = "file:///C:/Mobius/build/runtime/data/people.json";
const string dbName = "SampleHiveDataBaseForMobius";
const string tableName = "people";
var builder = SparkSession.Builder().EnableHiveSupport();
// The following setting is required to use Spark 2.0 in Windows
// It may be provided in command line when running Mobius app
//builder = builder.Config("spark.sql.warehouse.dir", "<hdfs or local path>");
var session = builder.GetOrCreate();
var peopleDataFrame = session.Read().Json(jsonFilePath);
session.Sql(string.Format("CREATE DATABASE IF NOT EXISTS {0}", dbName)); // create database if not exists
session.Sql(string.Format("USE {0}", dbName));
//hiveContext.Sql(string.Format("DROP TABLE {0}", tableName)); // drop table if exists
peopleDataFrame.Write().Mode(SaveMode.Overwrite).SaveAsTable(tableName); // create table
var tablesDataFrame = session.Table(tableName); // get all tables in database
logger.LogInfo(string.Format("table count in database {0}: {1}", dbName, tablesDataFrame.Count()));
tablesDataFrame.Show();
session.Sql(string.Format("SELECT * FROM {0}", tableName)).Show(); // select from table
I am getting error @
var peopleDataFrame = session.Read().Json(jsonFilePath);
Source: Microsoft.Spark.CSharp.Adapter
Message: JVM method execution failed: Nonstatic method load failed for class 14 when called with no parameters
And here is the stack trace
Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] parameters)
Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference objectId, String methodName)
Microsoft.Spark.CSharp.Proxy.Ipc.DataFrameReaderIpcProxy.Load()
Microsoft.Spark.CSharp.Sql.DataFrameReader.Load()
Microsoft.Spark.CSharp.Sql.DataFrameReader.Load(String path)
Microsoft.Spark.CSharp.Sql.DataFrameReader.Json(String path)
DemoSparkHiveDataFrame.Program.Main(String[] args) in e:\Work\Feb\VS 2012\DemoSparkHiveDataFrame\DemoSparkHiveDataFrame\Program.cs:line 26
System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
System.Threading.ThreadHelper.ThreadStart_Context(Object state)
System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
System.Threading.ThreadHelper.ThreadStart()
I got a solution i just need to set
builder.config("spark.sql.warehouse.dir","file:///C:/spark-warehouse")
for set spark.sql.warehouse.dir path and its works fine for me.