Search code examples
hdfscluster-computinghadoop2federation

Namenodes don't use all datanodes in a fully distributed hadoop cluster with HDFS Federation

Actually working on a Big Data project as a final project assignement, I'v been looking for a mean to run HDFS Federation on a fully distributed cluster.

The specifications of my cluster are :

  • Hadoop 2.7.2
  • JDK 1.8.74
  • The OS system is CentOS 6.7
  • 2 namenodes (Namenode1 and Namenode2)
  • 2 datanodes (Datanode1 and Datanode2)
  • 1 Client (configured for ViewFS mount table)

With one namenode, the cluster (1 namenode + 2 datanodes) works fine, all the configurations seem correct.

I couldn't find that many tutorials explaning how to fully configure the HDFS Federation (for running two namenodes that share all the datanodes) not even in the official documentation of Apache Hadoop. The one I used is the following Fully Distributed Hadoop Federation Cluster

My attempts to run effectively the HDFS Federation have failed, even if the dfs daemons have successfully launched, the datanodes are not used by all the namenodes.

Actual situation :

When I start the dfs services (with start-dfs.sh), the Namednode1 use all the datanodes and the Namenode2 use none. Or each namenode use only one unique datanode (Namenode1 uses Datanode1 and Namenode2 uses Datanode2).

The datanodes usage seems random but they are never all used by the two namenodes at the same time (my objective)

If anyone knows how to run the HDFS Federation with several namenodes, you're welcome to help =P Thank you.

Solution

  • I've finally found the issue and corrected it.

    It was in the hdfs-site.xml configuration file of the namenodes. You find it within the hadoop installation repertory : in my case it is located in ~/hadoop-2.7.2/etc/hadoop/

    In addition the standard configuration, make sure that the dfs.name.dir property refers to the location of the metadata repertory of each namenode. In my case, for all the namenodes I created the /hdfs/name repertory for that purpose with the appropriate ownership (a user specially created for using hadoop)

    So you have to add in the hdfs-site.xml of all the namenodes of you cluster the following lines il addition of the other configuration :

    <property>
<name>dfs.name.dir</name>
<value>file:///hdfs/name,file:///hdfs/name</value>
</property>

    Now both the namenodes of my cluster use all the datanodes. Hope it will be helpful. Also I recommend to read this book which is very useful to correct the issue I faced Hadoop Operations - Eric Sammer (O'Reilly)