I'm running the build for Hadoop from the AUR and am getting a build failure that has something to do with hadoop-hdfs-native.
Here's the final failure:
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.3.5:cmake-compile (cmake-compile) on project hadoop-hdfs-native-client: make failed with error code 2 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.3.5:cmake-compile (cmake-compile) on project hadoop-hdfs-native-client: make failed with error code 2
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:375)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:298)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.MojoExecutionException: make failed with error code 2
at org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runMake (CompileMojo.java:229)
at org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.execute (CompileMojo.java:98)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:298)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :hadoop-hdfs-native-client
And here's a sample from some of the logs right before the failure:
[WARNING] make[2]: *** [main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/build.make:146: main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/configuration_loader.cc.o] Error 1
[WARNING] /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/sasl_digest_md5.cc: In function ‘std::string hdfs::GetMD5Digest(const std::string&)’:
[WARNING] /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/sasl_digest_md5.cc:225:11: warning: ‘int MD5_Init(MD5_CTX*)’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
[WARNING] 225 | MD5_Init(&ctx);
[WARNING] | ~~~~~~~~^~~~~~
[WARNING] In file included from /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/sasl_digest_md5.cc:23:
[WARNING] /usr/include/openssl/md5.h:49:27: note: declared here
[WARNING] 49 | OSSL_DEPRECATEDIN_3_0 int MD5_Init(MD5_CTX *c);
[WARNING] | ^~~~~~~~
[WARNING] /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/sasl_digest_md5.cc:226:13: warning: ‘int MD5_Update(MD5_CTX*, const void*, size_t)’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
[WARNING] 226 | MD5_Update(&ctx, src.c_str(), src.size());
[WARNING] | ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[WARNING] /usr/include/openssl/md5.h:50:27: note: declared here
[WARNING] 50 | OSSL_DEPRECATEDIN_3_0 int MD5_Update(MD5_CTX *c, const void *data, size_t len);
[WARNING] | ^~~~~~~~~~
[WARNING] /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/sasl_digest_md5.cc:227:12: warning: ‘int MD5_Final(unsigned char*, MD5_CTX*)’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
[WARNING] 227 | MD5_Final(reinterpret_cast<unsigned char *>(res), &ctx);
[WARNING] | ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[WARNING] /usr/include/openssl/md5.h:51:27: note: declared here
[WARNING] 51 | OSSL_DEPRECATEDIN_3_0 int MD5_Final(unsigned char *md, MD5_CTX *c);
[WARNING] | ^~~~~~~~~
[WARNING] make[2]: *** [main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/build.make:160: main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/hdfs_configuration.cc.o] Error 1
[WARNING] /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc: In function ‘std::string hdfs::SerializeDelimitedProtobufMessage(const google::protobuf::MessageLite*, bool*)’:
[WARNING] /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:59:27: warning: ‘int google::protobuf::MessageLite::ByteSize() const’ is deprecated: Please use ByteSizeLong() instead [-Wdeprecated-declarations]
[WARNING] 59 | int size = msg->ByteSize();
[WARNING] | ~~~~~~~~~~~~~^~
[WARNING] In file included from /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:22:
[WARNING] /usr/include/google/protobuf/message_lite.h:381:7: note: declared here
[WARNING] 381 | int ByteSize() const { return internal::ToIntSize(ByteSizeLong()); }
[WARNING] | ^~~~~~~~
[WARNING] /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc: In function ‘int hdfs::DelimitedPBMessageSize(const google::protobuf::MessageLite*)’:
[WARNING] /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:72:30: warning: ‘int google::protobuf::MessageLite::ByteSize() const’ is deprecated: Please use ByteSizeLong() instead [-Wdeprecated-declarations]
[WARNING] 72 | size_t size = msg->ByteSize();
[WARNING] | ~~~~~~~~~~~~~^~
[WARNING] /usr/include/google/protobuf/message_lite.h:381:7: note: declared here
[WARNING] 381 | int ByteSize() const { return internal::ToIntSize(ByteSizeLong()); }
[WARNING] | ^~~~~~~~
[WARNING] /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc: In function ‘std::string hdfs::GetRandomClientName()’:
[WARNING] /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:78:20: warning: ‘int RAND_pseudo_bytes(unsigned char*, int)’ is deprecated: Since OpenSSL 1.1.0 [-Wdeprecated-declarations]
[WARNING] 78 | RAND_pseudo_bytes(&buf[0], 8);
[WARNING] | ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
[WARNING] In file included from /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.h:28,
[WARNING] from /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc:19:
[WARNING] /usr/include/openssl/rand.h:79:29: note: declared here
[WARNING] 79 | OSSL_DEPRECATEDIN_1_1_0 int RAND_pseudo_bytes(unsigned char *buf, int num);
[WARNING] | ^~~~~~~~~~~~~~~~~
[WARNING] make[1]: *** [CMakeFiles/Makefile2:1924: main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/all] Error 2
[WARNING] /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/gmock-1.7.0/gmock-gtest-all.cc: In function ‘bool testing::internal::StackGrowsDown()’:
[WARNING] /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/gmock-1.7.0/gmock-gtest-all.cc:7485:24: warning: ‘dummy’ may be used uninitialized [-Wmaybe-uninitialized]
[WARNING] 7485 | StackLowerThanAddress(&dummy, &result);
[WARNING] | ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
[WARNING] /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/gmock-1.7.0/gmock-gtest-all.cc:7477:6: note: by argument 1 of type ‘const void*’ to ‘void testing::internal::StackLowerThanAddress(const void*, bool*)’ declared here
[WARNING] 7477 | void StackLowerThanAddress(const void* ptr, bool* result) {
[WARNING] | ^~~~~~~~~~~~~~~~~~~~~
[WARNING] /home/dominik/software/hadoop/src/hadoop-rel-release-3.3.5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/gmock-1.7.0/gmock-gtest-all.cc:7483:7: note: ‘dummy’ declared here
[WARNING] 7483 | int dummy;
[WARNING] | ^~~~~
[WARNING] make: *** [Makefile:146: all] Error 2
I think it may have something to do with a C++11/17 issue, but the steps needed to fix it are unclear to me.
Eventually my question on AUR got a response. The solution was to modify the PKGBUILD so that the build function was now
build() {
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk
export CXXFLAGS="$CXXFLAGS -include cstdint" # added line
cd hadoop-rel-release-${pkgver}
mvn package -Pdist,native \
-Drequire.openssl -Drequire.zstd \
-Dmaven.javadoc.skip=true -DskipTests
}
This fixed the issue and let the full build finish.