I finally managed to get spark running locally and start it (?) using:
spark-shell
I see this like starting a server program that can be consumed via APIs - not sure if this is correct? Or is the spark master running anyway? The result is this print out:
Spark context Web UI available at http://123.134.0.4:4040
Spark context available as 'sc' (master = local[*], app id = local-1492347943688
).
Spark session available as 'spark'.
I can access the web ui but how do I parameterise the sparkcontext in my C# driver program to run against the local spark. The ultimate aim is to run this Scala code in C#:
scala> val file = sc.textFile("C:\\war_and_peace.txt")
scala> val warsCount = file.filter(line => line.contains("war"))
scala> val peaceCount = file.filter(line => line.contains("peace"))
scala> warsCount.count()
res0: Long = 1218
scala> peaceCount.count()
res1: Long = 128
and get the word counts back to write into the console for example. Any pointers would be very much appreciated. Thanks!
PS:
It appears that I have to create an executable of the code and then use: sparkclr-submit.cmd or sparkclr-submit.sh to execute it. Is this correct? If so it means I cannot run/debug the code in Visual studio?!
Spark-shell.cmd does not launch a local Spark cluster. The shell is used to connect to an existing Spark cluster (YARN, Standalone or Mesos) or Spark in local mode. Refer to Spark documentation on providing appropriate 'master' configuration (http://spark.apache.org/docs/latest/submitting-applications.html#master-url) when running spark-submit.cmd. You need to provide the same set of parameters to sparkclr-submit.cmd in Mobius. The only additional parameter you need to provide is the exe name. To learn how to implement Word count app in Mobius refer to the example at https://github.com/Microsoft/Mobius/blob/master/notes/running-mobius-app.md#wordcount-example-batch
If you want to debug in VS, follow the debug mode instructions at https://github.com/Microsoft/Mobius/blob/master/notes/running-mobius-app.md#debug-mode