Search code examples
snakemake

Is there a way to get a nice error report summary when running many jobs on DRMAA cluster?

I need to run a snakemake pipeline on a DRMAA cluster with a total number of >2000 jobs. When some of the jobs have failed, I would like to receive in the end an easy readable summary report, where only the failed jobs are listed instead of the whole job summary as given in the log.

Is there a way to achieve this without parsing the log file by myself?

These are the (incomplete) cluster options:

jobs: 200 latency-wait: 5 keep-going: True rerun-incomplete: True restart-times: 2

Solution

  • I am not sure if there is another way than parsing the log file yourself, but I've done it several times with grep and I am happy with the results:

    cat .snakemake/log/[TIME].snakemake.log | grep -B 3 -A 3 error

    Of course you should change the TIME placeholder for whichever run you want to check.