I am running Oozie inside a custom MapR sandbox. Below is my job.properties file information:
nameNode=maprfs:///
jobTracker=maprfs:///
queueName=default
examplesRoot=examples
oozie.wf.application.path=/user/user01/lab2/exercise4/zayd_hammoudeh_lab2_exercise4/workflow.xml
For reference, below is my workflow.xml file:
<action name="stdev-calc">
<map-reduce>
<job-tracker>${jobTracker}</job-tracker>
<name-node>"${nameNode}"</name-node>
<prepare>
<delete path="/user/user01/lab2/exercise4/OUT"/>
</prepare>
<configuration>
<property>
<name>mapred.mapper.class</name>
<value>Stdev.StdevMapper</value>
</property>
<property>
<name>mapred.reducer.class</name>
<value>Stdev.StdevReducer</value>
</property>
<property>
<name>mapred.input.dir</name>
<value>/user/user01/lab2/exercise4/DATA</value>
</property>
<property>
<name>mapred.output.dir</name>
<value>/user/user01/lab2/exercise4/OUT</value>
</property>
</configuration>
</map-reduce>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Exercise 4 failed with error message: [${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
When I try to run my job, I get the following in my job info:
Job ID : 0000004-151018210741238-oozie-mapr-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : aggregator-wf
App Path : /user/user01/lab2/exercise4/zayd_hammoudeh_lab2_exercise4/workflow.xml
Status : KILLED
Run : 0
User : user01
Group : -
Created : 2015-10-19 04:42 GMT
Started : 2015-10-19 04:42 GMT
Last Modified : 2015-10-19 04:42 GMT
Ended : 2015-10-19 04:42 GMT
CoordAction ID: -
Actions
------------------------------------------------------------------------------------------------------------------------------------
ID Status Ext ID Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
0000004-151018210741238-oozie-mapr-W@:start: OK - OK -
------------------------------------------------------------------------------------------------------------------------------------
0000004-151018210741238-oozie-mapr-W@stdev-calc ERROR - ERROR IllegalArgumentException
------------------------------------------------------------------------------------------------------------------------------------
0000004-151018210741238-oozie-mapr-W@fail OK - OK E0729
------------------------------------------------------------------------------------------------------------------------------------
My logs show the error as: "[IllegalArgumentException: Illegal character in scheme name at index 0: "maprfs:///"]" For reference, below is my log:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/mapr/oozie/oozie-4.1.0/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/mapr/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
2015-10-18 21:42:06,368 INFO ActionStartXCommand:541 - SERVER[mapr1node] USER[user01] GROUP[-] TOKEN[] APP[aggregator-wf] JOB[0000004-151018210741238-oozie-mapr-W] ACTION[0000004-151018210741238-oozie-mapr-W@:start:] Start action [0000004-151018210741238-oozie-mapr-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2015-10-18 21:42:06,369 INFO ActionStartXCommand:541 - SERVER[mapr1node] USER[user01] GROUP[-] TOKEN[] APP[aggregator-wf] JOB[0000004-151018210741238-oozie-mapr-W] ACTION[0000004-151018210741238-oozie-mapr-W@:start:] [***0000004-151018210741238-oozie-mapr-W@:start:***]Action status=DONE
2015-10-18 21:42:06,372 INFO ActionStartXCommand:541 - SERVER[mapr1node] USER[user01] GROUP[-] TOKEN[] APP[aggregator-wf] JOB[0000004-151018210741238-oozie-mapr-W] ACTION[0000004-151018210741238-oozie-mapr-W@:start:] [***0000004-151018210741238-oozie-mapr-W@:start:***]Action updated in DB!
2015-10-18 21:42:06,413 INFO ActionStartXCommand:541 - SERVER[mapr1node] USER[user01] GROUP[-] TOKEN[] APP[aggregator-wf] JOB[0000004-151018210741238-oozie-mapr-W] ACTION[0000004-151018210741238-oozie-mapr-W@stdev-calc] Start action [0000004-151018210741238-oozie-mapr-W@stdev-calc] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2015-10-18 21:42:06,468 WARN ActionStartXCommand:544 - SERVER[mapr1node] USER[user01] GROUP[-] TOKEN[] APP[aggregator-wf] JOB[0000004-151018210741238-oozie-mapr-W] ACTION[0000004-151018210741238-oozie-mapr-W@stdev-calc] Error starting action [stdev-calc]. ErrorType [ERROR], ErrorCode [IllegalArgumentException], Message [IllegalArgumentException: Illegal character in scheme name at index 0: "maprfs:///"]
org.apache.oozie.action.ActionExecutorException: IllegalArgumentException: Illegal character in scheme name at index 0: "maprfs:///"
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:401)
at org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:532)
at org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:666)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:884)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1134)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
at org.apache.oozie.command.XCommand.call(XCommand.java:281)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Illegal character in scheme name at index 0: "maprfs:///"
at java.net.URI.create(URI.java:859)
at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:177)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:392)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:310)
at org.apache.oozie.util.JobUtils.addFileToClassPath(JobUtils.java:151)
at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:539)
at org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:536)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
at org.apache.oozie.service.HadoopAccessorService.addFileToClassPath(HadoopAccessorService.java:536)
at org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:512)
... 12 more
Caused by: java.net.URISyntaxException: Illegal character in scheme name at index 0: "maprfs:///"
at java.net.URI$Parser.fail(URI.java:2829)
at java.net.URI$Parser.checkChars(URI.java:3002)
at java.net.URI$Parser.checkChar(URI.java:3012)
at java.net.URI$Parser.parse(URI.java:3028)
at java.net.URI.<init>(URI.java:595)
at java.net.URI.create(URI.java:857)
... 23 more
2015-10-18 21:42:06,468 WARN ActionStartXCommand:544 - SERVER[mapr1node] USER[user01] GROUP[-] TOKEN[] APP[aggregator-wf] JOB[0000004-151018210741238-oozie-mapr-W] ACTION[0000004-151018210741238-oozie-mapr-W@stdev-calc] Setting Action Status to [DONE]
2015-10-18 21:42:06,494 INFO ActionEndXCommand:541 - SERVER[mapr1node] USER[user01] GROUP[-] TOKEN[] APP[aggregator-wf] JOB[0000004-151018210741238-oozie-mapr-W] ACTION[0000004-151018210741238-oozie-mapr-W@stdev-calc] ERROR is considered as FAILED for SLA
2015-10-18 21:42:06,518 INFO ActionStartXCommand:541 - SERVER[mapr1node] USER[user01] GROUP[-] TOKEN[] APP[aggregator-wf] JOB[0000004-151018210741238-oozie-mapr-W] ACTION[0000004-151018210741238-oozie-mapr-W@fail] Start action [0000004-151018210741238-oozie-mapr-W@fail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2015-10-18 21:42:06,518 INFO ActionStartXCommand:541 - SERVER[mapr1node] USER[user01] GROUP[-] TOKEN[] APP[aggregator-wf] JOB[0000004-151018210741238-oozie-mapr-W] ACTION[0000004-151018210741238-oozie-mapr-W@fail] [***0000004-151018210741238-oozie-mapr-W@fail***]Action status=DONE
2015-10-18 21:42:06,518 INFO ActionStartXCommand:541 - SERVER[mapr1node] USER[user01] GROUP[-] TOKEN[] APP[aggregator-wf] JOB[0000004-151018210741238-oozie-mapr-W] ACTION[0000004-151018210741238-oozie-mapr-W@fail] [***0000004-151018210741238-oozie-mapr-W@fail***]Action updated in DB!
"
Tried everything I could to figure out what may be the cause of the issue here with no success. Using "maprfs:///" for the namenode and jobtracker is recommended by the examples that come bundled with the mapr-oozie package. Any help would be really appreciated.
The issue that I missed earlier is that when I was getting the error on the name node, it was not due to the contents of the "job.properties" file. Rather, it was coming from the "workflow.xml" file. I had mistakenly written:
<name-node>"${nameNode}"</name-node>
Oozie does not process the quotation marks, and if they are removed, the problem goes away. Key lesson in the end is to be careful to check both the workflow.xml and job.properties files when you see Oozie errors that are not easily explainable.