Search code examples
hadoopamazon-web-serviceselastic-map-reduce

Amazon ElasticMapReduce(EMR) controlling split size / num of mappers


How can I change this configuration? For my application, a split size of 64/128 is too much for me,

and I would like to have a split size of 16 mb for example.

How can I do it?


Solution

  • You can change the default block size by setting fs.s3n.block.size. Please try like below in your code - jobConf.set("fs.s3n.block.size", value);

    Please refer the below links as well - http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-bootstrap.html

    http://s3.amazonaws.com/awsdocs/ElasticMapReduce/latest/emr-dg.pdf