Search code examples
hadoopamazon-web-servicesemr

How to change block size an dreplication factor in EMR with hadoop version 1.0.3

I am trying to change the block and replication factor to 64MB and 2.

I am using java SDk to create EMR clsuter. I add following bootstrap action:

    BootstrapActionConfig setBlockSize = bootStrapSetHadoopProperty("Set Block Size",
            "dfs.block.size=67108864");
    bootstrapActions.add(setBlockSize);
   BootstrapActionConfig setReplicationFactor =   bootStrapSetHadoopProperty("Set Replication Factor",
            "dfs.replication=2");
    bootstrapActions.add(setReplicationFactor);

But when when i check hdfs-site.xml file in master node the values for dfs.block.size and dfs.replication remins un-changed. Any idea on how to do this?

Solution

  • Finally i was able to set the bootstrap option:

    Set Replication Factor  s3://elasticmapreduce/bootstrap-actions/configure-hadoop    -h, dfs.replication=2
Set Block Size  s3://elasticmapreduce/bootstrap-actions/configure-hadoop    -h, dfs.block.size=67108864

    -s option is deprecated. This URL tell about the option to use while creating bootstrap option: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-bootstrap.html#PredefinedbootstrapActions_ConfigureHadoop