I am currently running jobs on a cluster with qsub and have imposed a wall time of 10 hours on the jobs. Sometimes jobs exceed this time and are therefore terminated, and sometimes jobs are terminated for other reasons. Either way, I would like to have a footer at the bottom of the qsub log file which would indicate when the job terminated (optionally with the status code indicating whether it completed successfully or got interrupted) and the total wall time. Putting something like:
date
command here
date
in the qsub file wouldn't work since the last date won't get executed if the job goes over time. Does anyone know a way around this?
The qacct command should give you that information. From the manpage:
The qacct utility scans the accounting data file (see accounting(5)) and produces a summary of information for wall-clock time, cpu-time, and system time
Otherwise, if you really need the information to be in the output file, you can try something like
trap "date; exit 2" 2 3 15
in your script so that the date command is executed whenever the script is terminated.