AWS EMR - job counters exceeded limit 120

Question

I have a hadoop code base that I inherited and which I'm trying to get running on EMR. But I'm running into issues with the job counters. I get an error saying that I'm exceeding the default limit of 120. I looked into my code and I see I have about 40 counters, and EMR adds another 30 internal counters, but that should still be within the 120 default limit.

I'm running on EMR AMI version 2.4.2, and Amazon 1.0.3 hadoop distribution.

• Is there a way to increase the limit? I saw More than 120 counters in hadoop . But I'm not sure how to set this up on EMR.
• Is there any way I can get more debug to figure out what is going on?

Answer 1

You can raise the counter limit with this configuration:

[
  {
    "Classification": "mapred-site",
    "Properties": {
      "mapreduce.job.counters.max:": "1024"
    }
  }
]

Here are Amazon's instructions on how to register those instructions with your cluster. (I'm not pasting it here directly because there are many ways to do it, depending on how you create and use your cluster.)