Search code examples
memorypbs

Find maximum memory used by a PBS job

After a job is finished, how can I know the maximum resident size it required at any given point while running?

(tried /usr/bin/time, but not installed on the server)

Thank you!

Solution

  • PBS MOM reports some statistics back and it gets recorded in the PBS server log.

    A handy utility called tracejob parses the logs to extract all entries related to a specific job given a job ID.

    For example after the job completion on PBS Pro 12.1 tracejob would return several lines including the following

    07/11/2014 16:37:27  S    Exit_status=0 resources_used.cpupercent=98
                          resources_used.cput=01:49:14 resources_used.mem=5368kb
                          resources_used.ncpus=1 resources_used.vmem=38276kb
                          resources_used.walltime=01:49:22

    Here 5368 kb would correspond to the maximum rss.

    Similarly on Torque 3.0.5

    07/15/2014 03:45:12  S    Exit_status=0 resources_used.cput=20:44:10
                          resources_used.mem=704692kb
                          resources_used.vmem=1110224kb
                          resources_used.walltime=20:44:30

    Here the maximum rss was 704692 kb