got the following problem: I have got a hadoop cluster (2.8.1, java 8) and my Resource Manager keeps breaking after about 30 seconds-1minute or when I try to add any files to hdfs.
I have got 3 VPS (Ubuntu 16.04.2 LTS), 1 for Namenode and 2 for Data Nodes. These are mostly for playing around so just have 20gb of space (which i believe should be enough to see some tiny effect of mapreduce working)
contents of my files:
/etc/hosts on each of the servers:
135.59.171.41 01-slave 01-slave
188.166.167.150 02-master 02-master
155.59.170.39 03-slave 03-slave
hdfs-site.xml on master
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/hadoop/store/hdfs/namenode</value>
</property>
</configuration>
hdfs-site.xml on slaves
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/hadoop/store/hdfs/datanode</value>
</property>
</configuration>
core-site.xml on master and slaves
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://stegosaurus-server-02-master:9000</value>
</property>
</configuration>
JAVA HOME is in hadoop-env all 3 servers)
# The java implementation to use.
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
.bashrc hadoop related all 3 servers)
# -- HADOOP ENVIRONMENT VARIABLES START -- #
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export PATH=$PATH:$JAVA_HOME/bin
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export CLASSPATH=$CLASSPATH:/usr/local/hadoop/lib/*:.
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_OPTS="$HADOOP_OPTS -Djava.security.egd=file:/dev/../dev/urandom"
mapred-site.xml contents (all 3 servers)
<?xml version="1.0"?>
<!-- mapred-site.xml -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>02-master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>02-master:19888</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Djava.security.egd=file:/dev/../dev/urandom</value>
</property>
</configuration>
yarn-site.xml contents (all 3 servers)
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>02-master:8025</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>02-master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>02-master:8051</value>
</property>
</configuration>
I am using port 51 above because i was trying out all the different ports, including standard ones.
looking at the ports gives the following results
02-master:/usr/local/hadoop/etc/hadoop$ sudo lsof -i -P -n | grep LISTEN
sshd 1310 root 3u IPv4 13858 0t0 TCP *:22 (LISTEN)
sshd 1310 root 4u IPv6 13871 0t0 TCP *:22 (LISTEN)
apache2 1452 root 3u IPv4 15075 0t0 TCP *:80 (LISTEN)
apache2 1455 www-data 3u IPv4 15075 0t0 TCP *:80 (LISTEN)
apache2 1456 www-data 3u IPv4 15075 0t0 TCP *:80 (LISTEN)
apache2 1457 www-data 3u IPv4 15075 0t0 TCP *:80 (LISTEN)
apache2 1458 www-data 3u IPv4 15075 0t0 TCP *:80 (LISTEN)
apache2 1459 www-data 3u IPv4 15075 0t0 TCP *:80 (LISTEN)
java 7491 hduser 210u IPv4 170841 0t0 TCP *:50070 (LISTEN)
java 7491 hduser 226u IPv4 171053 0t0 TCP 188.166.169.154:9000
(LISTEN)
java 7718 hduser 220u IPv4 171925 0t0 TCP *:50090 (LISTEN)
apache2 19108 www-data 3u IPv4 15075 0t0 TCP *:80 (LISTEN)
masters and slaves are filled only on master: masters
02-master
slaves
01-slave
03-slave
Firewall is also off on all 3 servers for now.
What i figured out is that when RM crashes, if i telnet master from master via port that is used by yarn it will be "connection refused"
If i try
yarn application -list
it will try to connect to resourcemanager address port and die in 10 attempts.
When it loads I can see the web app on ports 8088 and 50070 but then when yarn dies obviously 8088 is not available I am not sure what else i can add here. This is quite confusing now because i think i have tried anything now. Can you help me?
running jps on namenode (please note that resource manager starts and then dies when trying ):
7491 NameNode
7718 SecondaryNameNode
12091 Jps
the last lines in ResourceManager logs are usually
scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
2017-10-25 21:01:13,025 INFO org.apache.hadoop.ipc.Server: Starting Socket
Reader #1 for port 8033
2017-10-25 21:01:13,054 INFO
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol
org.apache.hadoop.yarn.server.api.ResourceManagerAdministrationProtocolPB to
the server
2017-10-25 21:01:13,055 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2017-10-25 21:01:13,055 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8033: starting
IP V6 is switched off
Please help.. :(
As mentioned by
@tk421 - the problem was lack of RAM - I have doubled RAM on datanodes and quadrupled ram on NameNode and it started working.
Thanks a lot for your help @tk421 and have a great day !