Search code examples
hadoopapache-sparkooziehortonworks-data-platformoozie-workflow

Hortonworks Oozie Spark Action - NullPointerException

I am running on HDP 2.5.3 with oozie 4.2.0. The spark action is set to run in yarn-client mode. The Spark Job is for getting the data from hive table, process it and store it in HDFS. But when I try submitting the Spark Application from Spark Action, I am getting NullPointerException.

workflow.xml

<workflow-app xmlns="uri:oozie:workflow:0.5" name="Spark_Test">
   <global>
      <job-tracker>${job_tracker}</job-tracker>
      <name-node>${name_node}</name-node>
   </global>
   <credentials>
      <credential name="hiveCredentials" type="hive2">
         <property>
            <name>hive2.jdbc.url</name>
            <value>${hive_beeline_server}</value>
         </property>
         <property>
            <name>hive2.server.principal</name>
            <value>${hive_kerberos_principal}</value>
         </property>
      </credential>
   </credentials>
   <start to="SparkTest" />
   <action name="SparkTest" cred="hiveCredentials">
      <spark xmlns="uri:oozie:spark-action:0.1">
         <job-tracker>${job_tracker}</job-tracker>
         <name-node>${name_node}</name-node>
         <master>yarn-client</master>
         <name>Spark Hive Example</name>
         <class>com.fbr.genjson.exec.GenExecJson</class>
         <jar>${jarPath}/fedebomrpt_genjson.jar</jar>
         <spark-opts>--jars /usr/hdp/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar,/usr/hdp/current/spark-client/lib/datanucleus-core-3.2.10.jar --files /etc/hive/conf/hive-site.xml --conf spark.sql.hive.convertMetastoreOrc=false --driver-memory 2g --executor-memory 16g --executor-cores 4 --conf spark.ui.port=5051 --queue fbr</spark-opts>
         <arg>${arg1}</arg>
         <arg>${arg2}</arg>
      </spark>
      <ok to="end" />
      <error to="fail" />
   </action>
   <kill name="fail">
      <message>Spark Java PatentCitation failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
   </kill>
   <end name="end" />
</workflow-app>

Exception:

SERVER[xxx.hpc.xx.com] USER[prxtcbrd] GROUP[-] TOKEN[] APP[Spark_Test] JOB[0004629-170625082345353-oozie-oozi-W] ACTION[0004629-170625082345353-oozie-oozi-W@SparkTest] Error starting action [SparkTest]. ErrorType [ERROR], ErrorCode [NullPointerException], Message [NullPointerException: null]
org.apache.oozie.action.ActionExecutorException: NullPointerException: null
    at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1202)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1373)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
    at org.apache.oozie.command.XCommand.call(XCommand.java:287)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
    at org.apache.oozie.action.hadoop.SparkActionExecutor.setupActionConf(SparkActionExecutor.java:85)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1091)
    ... 11 more

I dont know where I am doing mistake.. Do I need add any config xmls other than hive-site.xml ?

Solution

  • In your example you import jars, files(hive-site.xml). I think there is no need to import these thing oozie already import these thing. Can you check with below spark action I think it might solve your problem. 

    <action name="myfirstsparkjob" cred="hive_credentials">
    <spark xmlns="uri:oozie:spark-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <configuration>
            <property>
                <name>mapred.compress.map.output</name>
                <value>true</value>
            </property>
            <property>
                <name>mapred.job.queue.name</name>
                <value>${queueName}</value>
            </property>
        </configuration>
        <master>yarn</master>
        <mode>cluster</mode>
        <name>Spark Hive Example</name>
        <class>com.fbr.genjson.exec.GenExecJson</class>
        <jar>${jarPath}/fedebomrpt_genjson.jar</jar>
        <spark-opts>--queue queue_name --executor-memory 28G --num-executors 70 --executor-cores 5</spark-opts>
    </spark>
  <ok to="end" />
  <error to="fail" />

    And also set below oozie properties in you workflow.xml file

    oozie.use.system.libpath=true oozie.libpath=${jarPath}

    Make sure you put all your user created libs and files inside your ${jarFile}