SparkClr: Sparkcontext gives an Error

Question

No connection could be made because the target machine actively refused it 127.0.0.1:some port number

Here is my code i have just wrote in console application in Programe.cs is

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Spark.CSharp.Core;

namespace DemoSpark
{
class Program
{
    static void Main(string[] args)
    {
        var Sparkcontext = new SparkContext(new SparkConf().SetAppName("Demo"));
        Sparkcontext.Stop();
        Console.WriteLine("Done");
    }
}
}

And the exception is there in Screenshot 1]: https://i.sstatic.net/FWKQo.png

Answer 1

You cannot directly run the executable Programe.cs. Instead it should be submitted to your spark cluster. First you must setup your environment to point to the Mobius source or release directory:

set SPARKCLR_EXT_JARS=C:\Code\Mobius\build\localmode\..\runtime\dependencies\spark-csv_2.10-1.4.0.jar,C:\Code\Mobius\build\localmode\..\runtime\dependencies\commons-csv-1.4.jar
set SPARKCLR_HOME=C:\Code\Mobius\build\localmode\..\runtime
set SPARK_HOME=C:\Code\Mobius\build\localmode\..\tools\spark-2.0.2-bin-hadoop2.6
set "JAVA_HOME=C:\Program Files\Java\jdk1.8.0_66"
set HADOOP_HOME=C:\Code\Mobius\build\localmode\..\tools\winutils

Next you need to submit the executable to spark. Here is an example of running samples in local mode:

C:\Code\Mobius\build\runtime\scripts\sparkclr-submit.cmd --jars %SPARKCLR_EXT_JARS% --conf spark.sql.warehouse.dir=C:\Tests --exe SparkCLRSamples.exe C:\Code\Mobius\build\runtime\samples --temp C:\Temp --data C:\Data\Mobius

Keep in mind that anything after the --exe option is used by the spark workers to invoke executable, including commandline arguments.

See the documentation for other cluster options.