Search code examples
apache-sparkhadoopoozie

How to fix ''java.lang.NoSuchMethodError" in oozie spark yarn submit?

I have a bare metal hadoop 3.1.2 cluster up and running and I've also installed Oozie 5.1.0 and managed to run shell example successfully. But when I'm running Spark example (FileCopy) it indicates 'Success' in oozie, while file isn't really copied, and YARN stderr contains:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse.getResourceProfiles()Ljava/util/Map;
at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:253)
at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:234)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:201)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.registerApplicationMaster(AMRMClientAsyncImpl.java:180)

Do you have any idea why this may happen?

Solution

  • Oozie 5.1.0 does not support Hadoop 3. Open issues: