I'm building a small Hadoop cluster composed of 2 nodes : 1 master + 1 worker. I'm using the latest version of Hadoop (3.2) and everything is executed by the root user. In the installation process, I've been able to hdfs namenode -format. Next step is to start the HDFS daemon with start-dfs.sh.
$ start-dfs.sh
Starting namenodes on [master]
bash v3.2+ is required. Sorry.
Starting datanodes
bash v3.2+ is required. Sorry.
Starting secondary namenodes [master]
bash v3.2+ is required. Sorry.
Here's the generated logs in the journal:
$ journalctl --since "1 min ago"
-- Logs begin at Thu 2019-08-29 11:12:27 CEST, end at Thu 2019-08-29 11:46:40 CEST. --
Aug 29 11:46:40 master su[3329]: (to root) root on pts/0
Aug 29 11:46:40 master su[3329]: pam_unix(su-l:session): session opened for user root by root(uid=0)
Aug 29 11:46:40 master su[3329]: pam_unix(su-l:session): session closed for user root
Aug 29 11:46:40 master su[3334]: (to root) root on pts/0
Aug 29 11:46:40 master su[3334]: pam_unix(su-l:session): session opened for user root by root(uid=0)
Aug 29 11:46:40 master su[3334]: pam_unix(su-l:session): session closed for user root
Aug 29 11:46:40 master su[3389]: (to root) root on pts/0
Aug 29 11:46:40 master su[3389]: pam_unix(su-l:session): session opened for user root by root(uid=0)
Aug 29 11:46:40 master su[3389]: pam_unix(su-l:session): session closed for user root
As I'm using Zsh (with Oh-my-Zsh), I logged into a bash console to give it a try. Sadly, I get the same result. In fact, this error happens for all sbin/start-*.sh scripts. However, the hadoop and yarn commands work like a charm.
Since I didn't find much information on this error on the Internet, here I am. Would be glad to have any advice!
Operating system info:
$ lsb_release -d
Description: Debian GNU/Linux 10 (buster)
$ uname -srm
Linux 4.19.0-5-amd64 x86_64
Available Java versions (tried with both):
$ update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 auto mode
* 1 /usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/bin/java 1081 manual mode
2 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode
Some ENV variables you might be interested in:
$ env
USER=root
LOGNAME=root
HOME=/root
PATH=/root/bin:/usr/local/bin:/usr/local/hadoop/bin:/usr/local/hadoop/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SHELL=/usr/bin/zsh
TERM=rxvt-unicode
JAVA_HOME=/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64
HADOOP_HOME=/usr/local/hadoop
HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
ZSH=/root/.oh-my-zsh
Output of the Hadoop executable:
$ hadoop version
Hadoop 3.2.0
Source code repository https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf
Compiled by sunilg on 2019-01-08T06:08Z
Compiled with protoc 2.5.0
From source with checksum d3f0795ed0d9dc378e2c785d3668f39
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-3.2.0.jar
My Zsh and Bash installation:
$ zsh --version
zsh 5.7.1 (x86_64-debian-linux-gnu)
$ bash --version
GNU bash, version 5.0.3(1)-release (x86_64-pc-linux-gnu)
# only available in a console using *bash*
$ echo ${BASH_VERSINFO[@]}
5 0 3 1 release x86_64-pc-linux-gnu
TL;DR: use a different user (e.g. hadoop) instead of root.
I found the solution but not the deep understanding on what is going on. Despite how sad I can be, here's the solution I found:
Running with root user:
$ start-dfs.sh
Starting namenodes on [master]
bash v3.2+ is required. Sorry.
Starting datanodes
bash v3.2+ is required. Sorry.
Starting secondary namenodes [master_bis]
bash v3.2+ is required. Sorry
Then I created a hadoop user and gave this user privileges on the Hadoop installation (R/W access). After logging in with this new user I have the following output for the command that caused me some troubles:
$ start-dfs.sh
Starting namenodes on [master]
Starting datanodes
Starting secondary namenodes [master_bis]
Moreover, I noticed that processes created by start-yarn.sh were not listed in the output of jps while using Java 11. Switching to Java 8 solved my problem (don't forget to update all $JAVA_HOME variables, both in /etc/environment and hadoop-env.sh).
Success \o/. However, I'd be glad to understand why the root user cannot do this. I know it's a bad habit to use root but in an experimental environment this is not of our interest to have a clean "close-to" production environment. Any information about this will be kindly appreciated :).