Search code examples
hadoophdfs

HDFS is not working?

I want to set up multi nodes hadoop on three virtual machines running Ubuntu operating system.

I set up everything right: I ssh from any of them to another. I configure hadoop files. 

enter code here
*This is core.site-xml*

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://Master1:54310</value>
</property>
</configuration>

This is hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>dfs.name.dir</name>
<value>/hadoop/name/</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/hadoop/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>

This is mapred.site-xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>Master1:54311</value> // this is my hostname for master node
</property>
</configuration>

![enter image description here][1]

command line:

 inside hadoop I typed on command line:
 bin/hadoop namenode -format
 bin/hadoop start-all.sh
 jps
 bin/hadoop dfsadmin -report
enter code here

the outputs in the pictures below: Thank you in advance and I am looking forward your help

enter image description here

enter image description here

enter image description here

enter image description here

enter image description here

Solution

  • Update the below property in hdfs-site.xml then try

    <property>
    <name>dfs.name.dir</name>
    <value>file:/home/user/hadoop/name/</value>
</property>
<property>
    <name>dfs.data.dir</name>
    <value>file:/home/user/hadoop/data</value>
</property>

    I checked the log, it should be path error. Change it to appropriate path as above or create two folders /home/user/hadoop/name and /home/user/hadoop/data and set its path in the hdfs-site.xml file.

    Log

    2015-06-24 02:57:19,997 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in dfs.data.dir: can not create directory: /hadoop/data
2015-06-24 02:57:19,997 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in dfs.data.dir are invalid.
2015-06-24 02:57:19,998 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: