Running some Pig jobs, I noticed the following line in the logs:
[main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
However, a Google search does not reveal anything about the meaning of the parameter mapred.job.reduce.markreset.buffer.percent. Does anybody know what it's for?
From the mapred-default.xml documentation:
The percentage of memory -relative to the maximum heap size- to be used for caching values when using the mark-reset functionality.
Note that this refers to a property named mapreduce.reduce.markreset.buffer.percent. There are two APIs within Hadoop, mapred and mapreduce. See this question for information about their differences.
I am not sure about this particular property, but my guess would be either that you are using an older version of Hadoop that has not updated the property's name, or the Pig developers made a mistake and typed "mapred" instead of "mapreduce" (and that's why you are finding that the property is not set). In either case, I think you can feel confident that it means what I have quoted from the docs.