Search code examples
hadoophdfsoozie

Oozie java-action does not include core-site.xml


When running an Oozie java action on a freshly installed Hadoop HDP 2.2.2.4, and for example tries to access hdfs it accesses the wrong filesystem: java.lang.IllegalArgumentException: Wrong FS: hdfs:/tmp/text.txt, expected: file:///

It can be fixed by included the core-site.xml in the Oozie action:

<file>hdfs:/path-to-core-site.xml-on-hdfs</file>

But what is the reason and what is the proper fix?


Solution

  • The reason of that the core-site.xml is not included in the class-path of the java-action is because the property mapreduce.application.classpath points to the wrong directory:

    <snip>/etc/hadoop/conf/secure
    

    It should point to

    <snip>/etc/hadoop/conf
    

    i.e, the full property should be something like, in mapred-site.xml:

    <property>
      <name>mapreduce.application.classpath</name>
      <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf</value>
    </property>