I have been trying to run a python script in spark(1.3.1.2.3) and I' am using oozie to schedule the spark job. I have a 3 node cluster running HDP 2.3 installed using Ambari 2.1.1.
I'm running into the following error when executing the job..
>>> Invoking Main class now >>>
Fetching child yarn jobs
tag id : oozie-9d5f396daac34b4a41fed946fac0472
Child yarn jobs are found -
Spark Action Main class : org.apache.spark.deploy.SparkSubmit
Oozie Spark action configuration
=================================================================
--master
yarn-client
--deploy-mode
client
--name
boxplot outlier
--class
/usr/hdp/current/spark-client/AnalyticsJar/boxplot_outlier.py
--executor-memory
1G
--driver-memory
1G
--executor-cores
4
--num-executors
2
--conf
spark.yarn.queue=default
--verbose
/usr/hdp/current/spark-client/AnalyticsJar/boxplot_outlier.py
=================================================================
>>> Invoking Spark class now >>>
Traceback (most recent call last):
File "/usr/hdp/current/spark-client/AnalyticsJar/boxplot_outlier.py", line 129, in <module>
main()
File "/usr/hdp/current/spark-client/AnalyticsJar/boxplot_outlier.py", line 60, in main
sc = SparkContext(conf=conf)
File "/hadoop/yarn/local/filecache/1314/spark-core_2.10-1.1.0.jar/pyspark/context.py", line 107, in __init__
File "/hadoop/yarn/local/filecache/1314/spark-core_2.10-1.1.0.jar/pyspark/context.py", line 155, in _do_init
File "/hadoop/yarn/local/filecache/1314/spark-core_2.10-1.1.0.jar/pyspark/context.py", line 201, in _initialize_context
File "/hadoop/yarn/local/filecache/1314/spark-core_2.10-1.1.0.jar/py4j/java_gateway.py", line 701, in __call__
File "/hadoop/yarn/local/filecache/1314/spark-core_2.10-1.1.0.jar/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s signer information does not match signer information of other classes in the same package
at java.lang.ClassLoader.checkCerts(ClassLoader.java:895)
at java.lang.ClassLoader.preDefineClass(ClassLoader.java:665)
at java.lang.ClassLoader.defineClass(ClassLoader.java:758)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.eclipse.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:136)
at org.eclipse.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:129)
at org.eclipse.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:98)
at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:98)
at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:89)
at org.apache.spark.ui.WebUI.attachPage(WebUI.scala:67)
at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:60)
at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:60)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.ui.WebUI.attachTab(WebUI.scala:60)
at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:66)
at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:60)
at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:42)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:223)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:53)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
at py4j.Gateway.invoke(Gateway.java:214)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
at py4j.GatewayConnection.run(GatewayConnection.java:207)
at java.lang.Thread.run(Thread.java:745)
Intercepting System.exit(1)
<<< Invocation of Main class completed <<<
Here is my workflow.xml file
<?xml version="1.0" encoding="UTF-8"?>
<workflow-app xmlns='uri:oozie:workflow:0.4' name='sparkjob'>
<start to='spark-process' />
<action name='spark-process'>
<spark xmlns='uri:oozie:spark-action:0.1'>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>oozie.launcher.mapred.job.queue.name</name>
<value>launcher2</value>
</property>
<property>
<name>oozie.service.SparkConfigurationService.spark.configurations</name>
<value>spark.eventLog.dir=hdfs://node1.analytics.tardis:8020/user/spark/applicationHistory,spark.yarn.historyServer.address=http://node1.analytics.tardis:18088,spark.eventLog.enabled=true</value>
</property>
</configuration>
<master>yarn-client</master>
<mode>client</mode>
<name>boxplot outlier</name>
<class>/usr/hdp/current/spark-client/AnalyticsJar/boxplot_outlier.py</class>
<jar>/usr/hdp/current/spark-client/AnalyticsJar/boxplot_outlier.py</jar>
<spark-opts>--executor-memory 1G --driver-memory 1G --executor-cores 4 --num-executors 2 --conf spark.yarn.queue=default</spark-opts>
</spark>
<ok to='end'/>
<error to='spark-fail'/>
</action>
<kill name='spark-fail'>
<message>Spark job failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name='end' />
</workflow-app>
From an initial search it seems that the error pops up because of conflicting dependencies in packaging a jar file that contains code for running the spark job. The python script boxplot_outlier.py does not import any dependency that might cause such a conflict.
In need of some guidance! Any suggestions would be greatly appreciated.
Edit: I checked the Classpath elements of the Oozie Java/Map-Reduce/Pig action launcher-job configuration and it includes the following two jars
/hadoop/yarn/local/usercache/ambari-qa/appcache/application_1441804290161_0903/container_e03_1441804290161_0903_01_000002/mr-framework/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar
/hadoop/yarn/local/usercache/ambari-qa/appcache/application_1441804290161_0903/container_e03_1441804290161_0903_01_000002/javax.servlet-3.0.0.v201112011016.jar
From the discussions in SPARK-1693 it seems that these two jars may cause this kind of a dependency conflict. The issue though has been stated as resolved in version 1.1.0 itself. There maybe an issue with the dependencies of hadoop 2.7 or there might be some configuration I'm missing. Any help would be appreciated
Finally solved it. Turns out removing the javax.servlet-3.0.0.v201112011016.jar from the oozie sharelib spark directory in hdfs mitigates the issue. I'm not sure whether this is the right way to tackle the issue and if this is a configuration issue with the HDP 2.3.0 distribution. Will report it to the HDP guys for further investigation.