Nextflow: How to limit number of total CPUs (or memory) used by local executor?

Question

With Nextflow, it is easy to set number of CPUs for each process. But how to limit number of TOTAL CPUs (or memory) available for the pipeline when using the local executor?

For example, when running the pipeline on a local server, I want to limit number of utilized CPUs. Having only one process definition that would be quite easy with e.g. maxForks. But what if pipeline consist of multiple processes that require different resources? How to enforce such limitation?

Minimal example can look like:

ch_processes1 = Channel.from( 1..10 )
ch_processes2 = Channel.from( 1..20 )

process cpuStress1 {
    
    cpus 3

    input:
    val(process)

    script:
    """
    stress --cpu ${task.cpus} --timeout 10
    """
}

process cpuStress2 {
    
    cpus 1

    input:
    val(process1)

    script:
    """
    stress --cpu ${task.cpus} --timeout 10
    """
}

workflow {
    cpuStress1(ch_processes1)
    cpuStress2(ch_processes2)
}

EDIT: I see that to some extend this can be achieved by taskset -c 0-2 nextflow run ..., where 0-2 defines range of CPU id (CPUs 0, 1, 2). But how to provide just NUMBER of CPUs without defining which one?

Answer 1

You can use executor scope definitions in your config file. See docs here. Specifically, use executor.cpus.