Search code examples
parallel-processingcpuram

GNU parallel: limit CPUs and RAM per Job

I have a code, for example

cat sample_name.list | parallel -j 5 --max-args=1 --progress --keep-order --results logs --joblog logs.txt echo {1}

I can not find an option that help me to limit the number of CPUs and amount of RAM assigned at each job.

With lscpu I have 12 CPUs and 16Gb RAM; I want to give at each job 2 CPUs and 1G RAM

Any help? Thanks!

Solution

  • I would use -j 50%.

    This will run 6 jobs in parallel on a 12 core machine. This way there will be 2 cores per jobs.

    It is unclear what you want to happen if a job uses more than 1 GB.

    Maybe --memsuspend 1G is what you are looking for?

    This will start suspending jobs when the free memory falls below 2 GB. If there is only 1 GB free, only a single job will be allowed to run. When the jobs free up more memory, the suspended jobs will be resumed.

    The idea here is that suspended jobs will be swapped out, thus freeing up memory.

    It is particularly useful if your program runs for a long time with low memory usage, but needs a lot of memory when finishing up. Here you optimally only want a single job to be in the finishing up state at a time.