Search code examples
hadoophdfsoozie

How does the example find the lib in Oozie best case?

According to the document of Oozie, I try to run a map-reduce example on Oozie. As everyone knows, 'workflow.xml' (and 'coordinator.xml') should be in HDFS.

Then input the command: oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run. And I also know the 'job.properties' should be in local file system.

But there are two things confused me:

1.why dose the jar or class variable in workflow.xml come from directory Lib of HDFS?

2.There is a picture showing the content of oozie-examples-4.3.1.jar. This jar is in HDFS, how can it import Lib? enter image description here

Forgive my poor English.

Solution

  • The highlighted red box is part of the Hadoop and Java default classpath. Any Java code that's ran within YARN, as part of MapReduce has access to the packages that appear when you run hadoop classpath command. By the way, mapred.* classes of Hadoop are almost all deprecated

    That's nothing to do with Oozie, per say, but Oozie extends the Hadoop classpath with the Oozie ShareLib, which must be explicitly enabled with a property file argument

    oozie.use.system.libpath=true

    And in addition to that classpath,, Oozie will send the ${wf.application.path}/lib directory to all running jobs