Search code examples
hadoophadoop2

Hadoop changing IP addresses


I have installed a Hadoop 2.2.0 (mini-cluster) in Ubuntu 14.04. There are three nodes, one master and two slaves. During the installation I had to type ip adresses of nodes into the /etc/hosts. My nodes get ip adresses dynamically, so I have to edit /etc/hosts files on all nodes in every machine start.

Do anyone have an idea to eliminate editing the hosts files?


Solution

  • I would strongly recomend a DHCP static (or manual) IP configuration for those nodes. This is because many other Apache related solutions may trouble you in the future (i.e. if you install Apache Zookeper). Also, this can make troubles if the nodes change ip while Hadoop is running.

    If you still persist on that, I would recommend you writting a script that runs the networks looking for the Hostnames and their ips (nmap). After that, updating the /etc/hosts. That solution is quite instable, since it can make troubles if it changes while the hadoop cluster is already up and running.

    Reference: http://hadoop-common.472056.n3.nabble.com/hadoop-conf-for-dynamically-changing-ips-td676599.html