This is a continuation of this question and this question on nested GNU Parallel. Utlimately what I want to achieve is to leave my Makefile untouched except by changing the SHELL= variable and distribute jobs using parallel across all my machines.
Is there a way to ensure that concurrent executions of GNU Parallel respect the --jobs clause specified in the outer invocation? Or some other way to get a limit on the total number of jobs across parallel invocations? For example: I'd like the inner slot in the output below to always be 1, i.e., the slot 1-2 on line three of the output violates the condition.
~• inner_par="parallel -I // --slotreplace '/%/' --seqreplace '/#/'"
~• cmd='echo id {#}-/#/, slot {%}-/%/, arg {}-//'
~• seq 2 | parallel -j 1 "seq {} | $inner_par $cmd"
id 1-1, slot 1-1, arg 1-1
id 2-1, slot 1-1, arg 2-1
id 2-2, slot 1-2, arg 2-2
~•
Are you looking for sem?
parallel -j 10 parallel -j 20 sem -j 30 --id myid mycmd
This will start 200 sems, but only run 30 mycmds in parallel.