Search code examples
hiveoozie

Oozie workflow: Hive action failed because of Tez

Running my script on a data node running the hive client tools is working. But when i schedule the hive script using Oozie than i get the Error as shown below.

I've set the tez.lib.uris in the tez-site.xml to hdfs:///apps/tez/,hdfs:///apps/tez/lib/

What I'm missing here?

Hive script:

  USE av_raw;

  LOAD DATA INPATH '${INPUT}' INTO TABLE alarms_stg;

  INSERT INTO TABLE alarms PARTITION (year, month)
  SELECT * FROM alarms_stg WHERE job_id = '${JOBID}';

Workflow action:

<!-- load processed data and store in hive -->
<action name="load-data">
    <hive xmlns="uri:oozie:hive-action:0.3">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <job-xml>hive-site.xml</job-xml>
        <script>load_data.hive</script>
        <param>INPUT=${complete}</param>
        <param>JOBID=${wf:actionData('stage-data')['hadoopJobs']}</param>
    </hive>

    <ok to="end"/>

    <error to="fail"/>
</action>

Error:

 Log Type: stderr

  Log Length: 3227
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/grid/5/hadoop/yarn/local/filecache/2418/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  log4j:ERROR Could not find value for key log4j.appender.CLA
  log4j:ERROR Could not instantiate appender named "CLA".
  log4j:ERROR Could not find value for key log4j.appender.CLA
  log4j:ERROR Could not instantiate appender named "CLA".

  Logging initialized using configuration in file:/grid/2/hadoop/yarn/local/usercache/hdfs/appcache/application_1417175595182_12259/container_1417175595182_12259_01_000002/hive-log4j.properties
  Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw exception, org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configurartion
  java.lang.RuntimeException: org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configurartion
  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:358)
  at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
  at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
  at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:316)
  at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:277)
  at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
  at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
  Caused by: org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configurartion
  at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:137)
  at org.apache.tez.client.TezSession.start(TezSession.java:105)
  at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:185)
  at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:123)
  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:356)
  ... 19 more
Solution

  • Please try to add tez.lib.uris=hdfs:///apps/tez/,hdfs:///apps/tez/lib/ in workflow.xml of your Oozie job

    e.g) workflow.xml

    <!-- load processed data and store in hive -->
<action name="load-data">
    <hive xmlns="uri:oozie:hive-action:0.3">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <job-xml>hive-site.xml</job-xml>
        <configuration>
            <property>
                <name>tez.lib.uris</name>
                <value>hdfs:///apps/tez/,hdfs:///apps/tez/lib/</value>
            </property>
        </configuration>
        <script>load_data.hive</script>
        <param>INPUT=${complete}</param>
        <param>JOBID=${wf:actionData('stage-data')['hadoopJobs']}</param>
    </hive>
    <ok to="end"/>
    <error to="fail"/>
</action>