I am moving a Snakemake workflow that I developed in an LSF cluster
environment (e.g. bsub) to a SGE cluster environment (e.g. qsub).
I have previously specified my jobs' memory requirements using the
resources keyword in rules, using the variable mem_mb as suggested
in the docs. I have also specified the number of threads I would like to have available for the job using the threads keyword. These values sometimes end up as parameters for my jobs themselves (the program I'm running allows me to specify it's maximum memory usage).
I am having difficulty transitioning my cluster submission
profile/scripts specification of memory requirements to qsub,
where memory requirements per job are specified per core, versus bsub, where memory requirements are specified per job. For example, I may have a multithreaded job that requires 64GB memory that I want to run on 8 cores:
bsub, this is (basically) specified by
bsub -M 65536 -n 8 {...}.qsub, I need to specify the memory per core (8192):
qsub -l h_vmem=8192M -l m_mem_free=8192M -pe smp 8 {...}.In the past, my bsub cluster command would use the values
from resources/threads for the job submission:
snakemake --cluster "bsub -M {resources.mem_mb} -n {threads}".
I was hoping I could do math in Snakemake's string formatting:
snakemake --cluster "qsub -l h_vmem={resources.mem_mb // threads} -pe smp {threads}". Unfortunately this doesn't work -- it tells me that mem_mb // threads isn't an element of resources.
I expect to be able to have a Snakefile that takes minimal account of my cluster environment. Right now, I am specifying resources to complete the job, but that should be relatively platform independent.
Is there an easy/standard way for resolving this difference that I can apply in order to extract/calculate this job information for my cluster submission command without adding a separate resources command? That is, I'd like to avoid:
# xxx is the memory requirements of the job in MB
mem_mb = xxx
# + 1 to guarantee that mem_mb_per_thread * threads >= mem_mb
mem_mb_per_thread = lambda wc, threads: (xxx // threads) + 1
Right now, I anticipate going into my Snakefiles and adding a redundant resources parameter for a specific cluster platform, which seems inappropriate. What are best practices for this case?
The answer is kind of in the Snakemake docs: a custom Snakemake profile to provide a custom Python job submission script. I wasn't able to find it in their example profiles, but a good example of how it could be done is implemented at broadinstitute/snakemake-broad-uger. I have since made an implementation for bsub or qsub (SGE) based clusters using the --cluster-sync Snakemake directive at jaicher/snakemake-sync-bq-sub.
In this context, we can use the Snakemake helper function snakemake.utils.read_job_properties to read in job properties from the jobscript (the last argument to the jobscript). This allows us to extract job name, wildcards, resources, and cluster parameters from the job script, with which we can do additional processing. Thus, a minimal solution is not far off from my original idea which I wanted to avoid:
import sys # for sys.argv (command-line arguments)
from snakemake.utils import read_job_properties # to obtain properties
# get path of jobscript
jobscript = sys.argv[-1]
# get job properties
job_info = read_job_properties(jobscript)
# get threads information
threads = job_info.get("threads", 1)
# get resources information
resources = job_info.get("resources", dict())
# get mem_mb information
mem_mb = resources.get("mem_mb", DEFAULT_VALUE) # define DEFAULT_VALUE elsewhere
mem_mb_per_thread = round(mem_mb / threads, 2) # memory per thread
# if bsub, resources subcommand is...
bsub_subcmd = f" -M {mem_mb} -n {threads}"
# if qsub, resources subcommand is...
qsub_subcmd = (
f" -l h_vmem={mem_mb_per_thread}M -l m_mem_free={mem_mb_per_thread}M"
f" -pe smp {threads}"
)
# run something like `bsub {qsub_subcmd} {other_params} {jobscript}`