I am trying to run a Spark job via oozie on my HDP 2.6.1 cluster. The Spark versions initially installed are 1.6.1 and 2.1.1. I added the 2.3.0 pre-built package (Hadoop 2.7 and later) version via sharelibupdate (by creating a directory in which I copied all spark 2.3.0 jars + Oozie jar) but am facing this error : ..././assembly/target/scala-2.11/jars' does not exist; make sure Spark is built.
I am runing spark in Spark cluster mode.
Did anyone ever face this problem ? Is it even possible to add a spark version like this ?
I found a solution to this. It is related to this Oozie known problem. I simply had to add a spark property spark.yarn.jars with the path to spark 2.3.0 jars