Search code examples
jobsslurmnextflow

set slurm to distribute jobs across nodes in nextflow

I am running a nextflow pipeline on a 3-node cluster. When I run the pipeline through slurm, it creates a high number of jobs, that I limit by using the executor.queueSize = X directive. However, what slurm does is to saturate node 1, then saturate node 2, then starts sending jobs to node 3. I'd like it to distribute the job list more evenly. I've tried a number of slurm commands, including

--spread-job
--ntasks-per-core=5
--distribution=cyclic 
-m cyclic=1 
--distribution=plane=5

But none does what I want, which is just to assign 1 job to N1, then 1 to N2, then 1 to N3, then 1 to N1 again etc.

Any ideas please?

Thanks in advance for your help.

Solution

  • As a user, you do not decide how your independent jobs are allocated with respect to one another. The --spread-job and --distribution=cyclic options decide how the allocation for a single job is built, and how tasks are mapped onto that allocation.

    To obtain the behaviour you want, the cluster must be configured with SelectTypeParameters=CR_LLN

    This option leads to fragmented resource and makes it more difficult to schedule large jobs, so it often is not the default choice for clusters.