How to add mapreduce.reduce.memory.mb property to EMR Cluster in Cloud Formation template?

Question

I've been taking a look on how to modify the default values that EMR gives to the cluster depending on the type of machine it is.

In my case, it's a pretty basic setup of a m4.large as master and c3.2xlarge as core and the same for the task. The value that EMR sets for this property in the Task is 2880 (from http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/TaskConfiguration_H2.html). However, I need to increase this default value.

I've been trying to change this default configuration in my Cloud Formation template but for some reason it doesn't accept it. This is what I'm adding to the CF template, I can't seem to find any examples of this online:

    "Configurations" : [{
      "Classification": "mapred-site",
      "Properties": {
        "mapreduce.reduce.memory.mb": "4096"
    }}]

Any help would be greatly appreciated!

Answer 1

I had a similar problem, the solution for me was the change the key name from Properties to ConfigurationProperties because the syntax in CF specifically seems to be a bit different than you'd use within EMR's config section.

Try this:

"Configurations": [
  {
      "Classification": "mapred-site",
      "ConfigurationProperties": {
        "mapreduce.reduce.memory.mb": "4096"
      },
      "Configurations": []
  }
]