Search code examples
sqoopoozie

oozie 4.3.0 build failed


I need to use sqoop-1.4.6 arguments to export hive table to mysql, where the hive table has two partitionings. but I found out that the oozie share lib on my hadoop cluster only supports sqoop-1.4.3. So I try to build and create share lib fits our version specs.

I modified the oozie pom.xml as the case of our cluster master node, but build failed if I made the following two changes:

  1. If I change the hive version, Ln 92, from <hive.version>1.2.0</hive.version> to <hive.version>2.1.0</hive.version>, I get the following error:

Blockquote

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Oozie Main .................................. SUCCESS [  0.991 s]
[INFO] Apache Oozie Hadoop Utils hadoop-2-4.3.0 ........... SUCCESS [  1.843 s]
[INFO] Apache Oozie Hadoop Distcp hadoop-2-4.3.0 .......... SUCCESS [  0.215 s]
[INFO] Apache Oozie Hadoop Auth hadoop-2-4.3.0 Test ....... SUCCESS [  0.432 s]
[INFO] Apache Oozie Hadoop Libs ........................... SUCCESS [  0.017 s]
[INFO] Apache Oozie Client ................................ SUCCESS [ 12.492 s]
[INFO] Apache Oozie Share Lib Oozie ....................... SUCCESS [  1.868 s]
[INFO] Apache Oozie Share Lib HCatalog .................... SUCCESS [  8.001 s]
[INFO] Apache Oozie Share Lib Distcp ...................... SUCCESS [  0.733 s]
[INFO] Apache Oozie Core .................................. FAILURE [ 16.672 s]
[INFO] Apache Oozie Share Lib Streaming ................... SKIPPED
[INFO] Apache Oozie Share Lib Pig ......................... SKIPPED
[INFO] Apache Oozie Share Lib Hive ........................ SKIPPED
[INFO] Apache Oozie Share Lib Hive 2 ...................... SKIPPED
[INFO] Apache Oozie Share Lib Sqoop ....................... SKIPPED
[INFO] Apache Oozie Examples .............................. SKIPPED
[INFO] Apache Oozie Share Lib Spark ....................... SKIPPED
[INFO] Apache Oozie Share Lib ............................. SKIPPED
[INFO] Apache Oozie Docs .................................. SKIPPED
[INFO] Apache Oozie WebApp ................................ SKIPPED
[INFO] Apache Oozie Tools ................................. SKIPPED
[INFO] Apache Oozie MiniOozie ............................. SKIPPED
[INFO] Apache Oozie Distro ................................ SKIPPED
[INFO] Apache Oozie ZooKeeper Security Tests .............. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 44.067 s
[INFO] Finished at: 2017-03-27T17:36:56+08:00
[INFO] Final Memory: 193M/1464M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:testCompile (default-testCompile) on project oozie-core: Compilation failure: Compilation failure:
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java:[817,47] error: cannot find symbol
[ERROR] variable firstLogEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java:[818,32] error: cannot find symbol
[ERROR] variable firstLogEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java:[819,78] error: cannot find symbol
[ERROR] variable firstLogEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/service/TestEventHandlerService.java:[213,46] error: cannot find symbol
[ERROR] variable logEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/service/TestEventHandlerService.java:[214,31] error: cannot find symbol
[ERROR] variable logEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/service/TestEventHandlerService.java:[215,81] error: cannot find symbol
[ERROR] variable logEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/service/TestEventHandlerService.java:[221,46] error: cannot find symbol
[ERROR] variable logEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/service/TestEventHandlerService.java:[222,31] error: cannot find symbol
[ERROR] variable logEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/service/TestEventHandlerService.java:[231,31] error: cannot find symbol
[ERROR] variable logEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/service/TestEventHandlerService.java:[239,31] error: cannot find symbol
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :oozie-core
  1. If I change the hbase version, Ln 85, from <hbase.version>0.94.27</hbase.version> to <hbase.version>1.1.1</hbase.version>, I get the following error:

Blockquote

[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Oozie Core 4.3.0
[INFO] ------------------------------------------------------------------------
Downloading: http://maven.aliyun.com/nexus/content/repositories/central/org/apache/hbase/hbase/1.1.1/hbase-1.1.1.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Oozie Main .................................. SUCCESS [  1.008 s]
[INFO] Apache Oozie Hadoop Utils hadoop-2-4.3.0 ........... SUCCESS [  1.869 s]
[INFO] Apache Oozie Hadoop Distcp hadoop-2-4.3.0 .......... SUCCESS [  0.094 s]
[INFO] Apache Oozie Hadoop Auth hadoop-2-4.3.0 Test ....... SUCCESS [  0.287 s]
[INFO] Apache Oozie Hadoop Libs ........................... SUCCESS [  0.026 s]
[INFO] Apache Oozie Client ................................ SUCCESS [ 14.418 s]
[INFO] Apache Oozie Share Lib Oozie ....................... SUCCESS [  2.331 s]
[INFO] Apache Oozie Share Lib HCatalog .................... SUCCESS [  2.478 s]
[INFO] Apache Oozie Share Lib Distcp ...................... SUCCESS [  0.728 s]
[INFO] Apache Oozie Core .................................. FAILURE [  0.755 s]
[INFO] Apache Oozie Share Lib Streaming ................... SKIPPED
[INFO] Apache Oozie Share Lib Pig ......................... SKIPPED
[INFO] Apache Oozie Share Lib Hive ........................ SKIPPED
[INFO] Apache Oozie Share Lib Hive 2 ...................... SKIPPED
[INFO] Apache Oozie Share Lib Sqoop ....................... SKIPPED
[INFO] Apache Oozie Examples .............................. SKIPPED
[INFO] Apache Oozie Share Lib Spark ....................... SKIPPED
[INFO] Apache Oozie Share Lib ............................. SKIPPED
[INFO] Apache Oozie Docs .................................. SKIPPED
[INFO] Apache Oozie WebApp ................................ SKIPPED
[INFO] Apache Oozie Tools ................................. SKIPPED
[INFO] Apache Oozie MiniOozie ............................. SKIPPED
[INFO] Apache Oozie Distro ................................ SKIPPED
[INFO] Apache Oozie ZooKeeper Security Tests .............. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24.879 s
[INFO] Finished at: 2017-03-27T18:01:13+08:00
[INFO] Final Memory: 110M/1420M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project oozie-core: Could not resolve dependencies for project org.apache.oozie:oozie-core:jar:4.3.0: Could not find artifact org.apache.hbase:hbase:jar:1.1.1 in alimaven (http://maven.aliyun.com/nexus/content/repositories/central/) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :oozie-core

I am using a near by maven central mirror with a proxy, but that shouldn't be the problem because I also tried the US mirror, result's the same. And I've done my best to look up documentations, related posts, with no luck so far.

Please, some one, enlighten me please.


Solution

  • I couldn't find a proper solution to the build problem, so I used an ugly workaround. First untar oozie-sharelib-4.3.0.tar.gz, replace the libs causing problems for my application, then use oozie-setup.sh sharelib create -fs hdfs://<HDFS master node>:9000 -locallib oozie-sharelib-4.3.0.tar.gz to update share lib.

    However this is also not the proper way, because then oozie asks for a seemingly endless list of jars.

    I finally went back to using shell action to execute sqoop from bash, I had to solve a very strange jar version collision problem, which is by all means out of the scope of this quetion.

    I hope the ugly workaroud may help someone in urgent situation.