Search code examples
hadoopfilesystemshdfshadoop-yarnfsck

Target Replicas is 10 but found 3 replica

How can I fixed this?

/tmp/hadoop-yarn/staging/ubuntu/.staging/job_1450038005671_0025/job.jar: Under replicated BP-938294433-10.0.1.190-1450037861153:blk_1073744219_3398. Target Replicas is 10 but found 3 replica(s).

I get this when I run hadoop fsck / in my master node. I assume I should change a .xml file in conf or something similar I just don't know which file to change.

Note that dfs.replication in hdfs-site.xml is already set to 3. I don't have dfs.replication.max in my hdfs-site.xml file.

Solution

  • The replication count for files submitted as part of your job (jars, etc) is controlled by the parameter mapreduce.client.submit.file.replication (or mapred.submit.replication in pre 2.4 clusters) in mapred-site.xml. You can adjust this down for clusters that are smaller than 10 nodes, or just ignore the message from fsck.

    FWIW, there is a JIRA for this but I doubt it will ever get worked.