The title makes the question pretty clear. Why are there two different sections in the jobtracker for completed and retired jobs?
Thanks.
Job retirement is a normal procedure from the jobtracker to persist the jobs to disk and clear up memory. You can read more details on the Cloudera blog:
Once a job is complete it is kept in memory (up to
mapred.jobtracker.completeuserjobs.maximum) and on disk as per the above. There is a configuration value that controls the overall retirement policy of completed jobs:Key:
mapred.jobtracker.retirejob.interval
Default: 24 * 60 * 60 * 1000 (1 day)
In other words, completed jobs are retired after one day by default. The check for jobs to be retired is done by default every minute and can be controlled with:Key:
mapred.jobtracker.retirejob.check
Default: 60 * 1000 (60s in msecs)
The check runs continually while the JobTracker is running. If a job is retired it is simply removed from the in-memory list of the JobTracker (it also removes all Tasks for the job etc.). Jobs are not retired under at least 1 minute (hardcoded in JobTracker.java) of their finish time. The retire call also removes the JobTracker Local (see above) file for the job. All that is left are the two files per retired job in the history directory (hadoop.job.history.location) plus – if enabled – the Per Job files (hadoop.job.history.user.location).