Search code examples
hadoophdfsreplication

How NameNode recognizes that the specific file replication is set value, than configured replication 3?


hdfs-site.xml:

dfs.replication value configured 3


Assuming that i set replication of an specific file to 2:

./bin/hadoop dfs -setrep -w 2 /path/to/file.txt

  1. When NameNode receives heartbeat from DataNode,

Will NameNode consider as specified file path/to/file.txt is in under replication as per the configured replication or not?

If not, how it 'll be?


Solution

  • First, I would like to attempt to restate your question for clarity, to make sure I understand:

    Will the NameNode consider a file that has been manually set to a replication factor lower than the default (dfs.replication) to be under-replicated?

    No. The NameNode stores the replication factor of each file separately in its metadata, even if the replication factor was not set explicitly by calling -setrep. By default, the metadata for each file will copy the replication factor as specified in dfs.replication (3 in your example). It may be overridden, such as by calling -setrep. When the NameNode checks if a file is under-replicated, it checks the exact replication factor stored in the metadata for that individual file, not dfs.replication. If the file's replication factor is 2, and there are 2 replicas of each of its blocks, then this is fine, and the NameNode will not consider it to be under-replicated.

    Your question also makes mention of heartbeating from the DataNodes, which I think means you're interested in how interactions between the DataNodes and NameNodes relate to replication. There is also another form of communication between DataNodes and NameNodes called block reports. The block reports are the means by which DataNodes tell the NameNodes which block replicas they store. The NameNode analyzes block reports from all DataNodes to determine if a block is either under-replicated or over-replicated. If a block is under-replicated (e.g. replication factor is 2, but there is only one replica), then the NameNode schedules re-replication work so that another DataNode makes a copy of the replica. If a block is over-replicated (e.g. replication factor is 3, but there are 4 replicas), then the NameNode schedules one of the replicas to be deleted, and eventually one of the DataNodes will delete it locally.