Search code examples
hadoopooziecloudera-cdh

Oozie Fails - Running MRv1 on CDH 5.4.2


I'm trying to run an oozie workflow the should execute MRv1 hadoop job.

Started with Cloudera QuickStart VM 5.4.2-0. Configured it to use MRv1 (appended at the bottom).

But the workflow fails:

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.MapReduceMain], main() threw exception, org/apache/hadoop/yarn/exceptions/YarnException
java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
    at org.apache.oozie.action.hadoop.MapReduceMain.run(MapReduceMain.java:58)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
    at org.apache.oozie.action.hadoop.MapReduceMain.main(MapReduceMain.java:38)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

The strange thing is that when I look at the classpath, it is loading the jar files from the right location; namely /usr/lib/hadoop-0.20-mapreduce

-----configurations----

  • Configured OOzie to use MapReduce instead of yarn:

    • Changed config for Oozie in CDH by changing: MapReduce Service from YARN to MapReduce
    • Changed config for Oozie in by setting oozie-site.xml valve value:

      <property> <name>oozie.service.HadoopAccessorService.jobTracker.whitelist</name> <value>quickstart.cloudera:8021</value> </property>

    • Changed tomcat config using alternatives: sudo alternatives --config oozie-tomcat-deployment -- chose tomcat-conf.http.mr1

  • Configured Hadoop to use MapReduce: sudo alternatives --config hadoop-conf -- chose conf.cloudera.mapreduce


Solution

  • I had forgotten to update the actions to use MR1:

    $ sudo -u oozie hadoop fs -rmr /user/oozie/share
    
    sudo oozie-setup sharelib create -fs <FS_URI> -locallib /usr/lib/oozie/oozie-sharelib-mr1