Search code examples
pythonresourcescluster-computingsnakemake

Resource requests through snakemake --resources vs. cluster submission


I am hoping to clear up some confusion I have about the snakemake resources keyword (documented here). If I am running jobs on a cluster, does this parameter make an actual memory request from the cluster scheduler for a specific job? Or, does it only make sure snakemake won't submit jobs if they would exceed the resource limits specified from the commandline.

I am primarily confused because in my snakemake job logs, it says I only have 100mb memory allocated as resources. But, in my cluster submission command (eg: qsub) I am requesting a couple gb of memory and the jobs do not run out of memory (which I predict they should if there was only 100mb of allocated memory).


Solution

  • Your link is broken, but I'll answer with "it depends". Specifically on your snakemake version and how you are integrating with the cluster. If you use the --cluster option, you have access to your resource object in the script. I use slurm so my argument looks like sbatch --mem={resources.mem_mb}M .... I've not made it to version 8 yet, so I'll defer to someone else on that.

    The amount reported in the logs is probably the default, and if you hard code in your memory to qsub it won't matter.