I've successfully run the example Spark workflow ("Copy a file by launching a Spark Java program") provided in the Hue Oozie workflow editor (in the Cloudera 5.5.1 QuickStart VM).
I'm now trying to run it manually using the oozie commandline tool:
oozie job -oozie http://localhost:11000/oozie -config job.properties -run
The workflow XML is basically unchanged - I have copied it to HDFS and have the following job.properties:
nameNode=hdfs://localhost:8020
jobTracker=localhost:8032
oozie.wf.application.path=/user/cloudera/workflows/spark-scala/spark-scala.xml
input=/user/hue/oozie/workspaces/data/sonnets.txt
output=here
The job is accepted and appears in the Hue web dashboard, but is killed after a few seconds, and the logs report:
Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SparkMain not found
What is the problem here?
Oozie doesn't include the libraries for the Spark action, by default - you need to add the following to the job.properties:
oozie.use.system.libpath=true
(Clicking on the previously successful Hue workflow in the Hue Dashboard, you can select the Configuration tab to see the properties that Hue has provided)