I need to use sqoop-1.4.6 arguments to export hive table to mysql, where the hive table has two partitionings. but I found out that the oozie share lib on my hadoop cluster only supports sqoop-1.4.3. So I try to build and create share lib fits our version specs.
I modified the oozie pom.xml as the case of our cluster master node, but build failed if I made the following two changes:
<hive.version>1.2.0</hive.version> to <hive.version>2.1.0</hive.version>, I get the following error:Blockquote
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Oozie Main .................................. SUCCESS [ 0.991 s]
[INFO] Apache Oozie Hadoop Utils hadoop-2-4.3.0 ........... SUCCESS [ 1.843 s]
[INFO] Apache Oozie Hadoop Distcp hadoop-2-4.3.0 .......... SUCCESS [ 0.215 s]
[INFO] Apache Oozie Hadoop Auth hadoop-2-4.3.0 Test ....... SUCCESS [ 0.432 s]
[INFO] Apache Oozie Hadoop Libs ........................... SUCCESS [ 0.017 s]
[INFO] Apache Oozie Client ................................ SUCCESS [ 12.492 s]
[INFO] Apache Oozie Share Lib Oozie ....................... SUCCESS [ 1.868 s]
[INFO] Apache Oozie Share Lib HCatalog .................... SUCCESS [ 8.001 s]
[INFO] Apache Oozie Share Lib Distcp ...................... SUCCESS [ 0.733 s]
[INFO] Apache Oozie Core .................................. FAILURE [ 16.672 s]
[INFO] Apache Oozie Share Lib Streaming ................... SKIPPED
[INFO] Apache Oozie Share Lib Pig ......................... SKIPPED
[INFO] Apache Oozie Share Lib Hive ........................ SKIPPED
[INFO] Apache Oozie Share Lib Hive 2 ...................... SKIPPED
[INFO] Apache Oozie Share Lib Sqoop ....................... SKIPPED
[INFO] Apache Oozie Examples .............................. SKIPPED
[INFO] Apache Oozie Share Lib Spark ....................... SKIPPED
[INFO] Apache Oozie Share Lib ............................. SKIPPED
[INFO] Apache Oozie Docs .................................. SKIPPED
[INFO] Apache Oozie WebApp ................................ SKIPPED
[INFO] Apache Oozie Tools ................................. SKIPPED
[INFO] Apache Oozie MiniOozie ............................. SKIPPED
[INFO] Apache Oozie Distro ................................ SKIPPED
[INFO] Apache Oozie ZooKeeper Security Tests .............. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 44.067 s
[INFO] Finished at: 2017-03-27T17:36:56+08:00
[INFO] Final Memory: 193M/1464M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:testCompile (default-testCompile) on project oozie-core: Compilation failure: Compilation failure:
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java:[817,47] error: cannot find symbol
[ERROR] variable firstLogEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java:[818,32] error: cannot find symbol
[ERROR] variable firstLogEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/sla/TestSLACalculatorMemory.java:[819,78] error: cannot find symbol
[ERROR] variable firstLogEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/service/TestEventHandlerService.java:[213,46] error: cannot find symbol
[ERROR] variable logEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/service/TestEventHandlerService.java:[214,31] error: cannot find symbol
[ERROR] variable logEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/service/TestEventHandlerService.java:[215,81] error: cannot find symbol
[ERROR] variable logEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/service/TestEventHandlerService.java:[221,46] error: cannot find symbol
[ERROR] variable logEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/service/TestEventHandlerService.java:[222,31] error: cannot find symbol
[ERROR] variable logEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/service/TestEventHandlerService.java:[231,31] error: cannot find symbol
[ERROR] variable logEntry of type LoggingEvent
[ERROR] /tmp/oozie-4.3.0/core/src/test/java/org/apache/oozie/service/TestEventHandlerService.java:[239,31] error: cannot find symbol
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :oozie-core
<hbase.version>0.94.27</hbase.version> to <hbase.version>1.1.1</hbase.version>, I get the following error:Blockquote
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Oozie Core 4.3.0
[INFO] ------------------------------------------------------------------------
Downloading: http://maven.aliyun.com/nexus/content/repositories/central/org/apache/hbase/hbase/1.1.1/hbase-1.1.1.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Oozie Main .................................. SUCCESS [ 1.008 s]
[INFO] Apache Oozie Hadoop Utils hadoop-2-4.3.0 ........... SUCCESS [ 1.869 s]
[INFO] Apache Oozie Hadoop Distcp hadoop-2-4.3.0 .......... SUCCESS [ 0.094 s]
[INFO] Apache Oozie Hadoop Auth hadoop-2-4.3.0 Test ....... SUCCESS [ 0.287 s]
[INFO] Apache Oozie Hadoop Libs ........................... SUCCESS [ 0.026 s]
[INFO] Apache Oozie Client ................................ SUCCESS [ 14.418 s]
[INFO] Apache Oozie Share Lib Oozie ....................... SUCCESS [ 2.331 s]
[INFO] Apache Oozie Share Lib HCatalog .................... SUCCESS [ 2.478 s]
[INFO] Apache Oozie Share Lib Distcp ...................... SUCCESS [ 0.728 s]
[INFO] Apache Oozie Core .................................. FAILURE [ 0.755 s]
[INFO] Apache Oozie Share Lib Streaming ................... SKIPPED
[INFO] Apache Oozie Share Lib Pig ......................... SKIPPED
[INFO] Apache Oozie Share Lib Hive ........................ SKIPPED
[INFO] Apache Oozie Share Lib Hive 2 ...................... SKIPPED
[INFO] Apache Oozie Share Lib Sqoop ....................... SKIPPED
[INFO] Apache Oozie Examples .............................. SKIPPED
[INFO] Apache Oozie Share Lib Spark ....................... SKIPPED
[INFO] Apache Oozie Share Lib ............................. SKIPPED
[INFO] Apache Oozie Docs .................................. SKIPPED
[INFO] Apache Oozie WebApp ................................ SKIPPED
[INFO] Apache Oozie Tools ................................. SKIPPED
[INFO] Apache Oozie MiniOozie ............................. SKIPPED
[INFO] Apache Oozie Distro ................................ SKIPPED
[INFO] Apache Oozie ZooKeeper Security Tests .............. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24.879 s
[INFO] Finished at: 2017-03-27T18:01:13+08:00
[INFO] Final Memory: 110M/1420M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project oozie-core: Could not resolve dependencies for project org.apache.oozie:oozie-core:jar:4.3.0: Could not find artifact org.apache.hbase:hbase:jar:1.1.1 in alimaven (http://maven.aliyun.com/nexus/content/repositories/central/) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :oozie-core
I am using a near by maven central mirror with a proxy, but that shouldn't be the problem because I also tried the US mirror, result's the same. And I've done my best to look up documentations, related posts, with no luck so far.
Please, some one, enlighten me please.
I couldn't find a proper solution to the build problem, so I used an ugly workaround. First untar oozie-sharelib-4.3.0.tar.gz, replace the libs causing problems for my application, then use oozie-setup.sh sharelib create -fs hdfs://<HDFS master node>:9000 -locallib oozie-sharelib-4.3.0.tar.gz to update share lib.
However this is also not the proper way, because then oozie asks for a seemingly endless list of jars.
I finally went back to using shell action to execute sqoop from bash, I had to solve a very strange jar version collision problem, which is by all means out of the scope of this quetion.
I hope the ugly workaroud may help someone in urgent situation.