Search code examples
javahadoopmapreduceoozie

How to test hadoop MapReduce java action separately which was part of an oozie workflow (with file tag)?

I am trying to test MapReduce job in java separately which is part of an oozie workflow. I am able to successfully test MapReduce job in java by using below command. 

hadoop jar /tmp/lib/Example.jar com.example.MRDriver mapper=com.example.DataTransformMapper

The challenge is when there is tag as part of oozie java action. Please advise how to simulate or test this MapReduce job with file tag.

<action name="JavaMRAction">
<java>
<job-xml>${hbase_site_xml}</job-xml>
<main-class>com.example.MRDriver</main-class>
<java-opts>-Dsun.security.krb5.debug=true</java-opts>
<arg>mapper=com.example.DataTransformMapper</arg>
<file>${hdfs_sharelib_hbase_path}hbase-client.jar#hbase-client.jar</file>
<file>${hdfs_sharelib_hbase_path}hbase-common.jar#hbase-common.jar</file> 
<file>${kerberos_keytab_file}#${kerberos_keytab_filename}</file>            
</java>
<ok to="NextAction" />
<error to="ErrorAction" />
</action>
Solution

  • Use the "-libjars" option of the hadoop jar command: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html#jar