Start n sub-processes in bash and wait till everything is finished

Question

I'd like to test out several ffmpeg settings and run multiple commands as 'sub-processes'. I would not like to launch more than 4 processes at a time, though. What is a good way to do this? Maybe something like trapping the process id's before running the process in the bg and waiting till they're no longer in the process list? or xargs?

Here's the script with no 'queuing':

#!/bin/bash

for crf in 10 15 20 23 25 30 35 ; do
    for vid_preset in medium slow veryslow; do
        ffmpeg -r 25 -i pics/pic_%04d.png -vcodec libx264 -crf $crf -vpre $vid_preset anim_crf$crf\_$vid_preset.mp4
    done
done

Answer 1

A very simple approach is to just run each ffmpeg process in the background, then wait for them to complete before going on to the next value of crf.

for crf in 10 15 20 23 25 30 35 ; do
    for vid_preset in medium slow veryslow; do
        ffmpeg -r 25 -i pics/pic_%04d.png -vcodec libx264 \
               -crf $crf -vpre $vid_preset anim_crf${crf}_${vid_preset}.mp4 &
    done
    wait
done

If you have parallel installed, you can use it to run all 21 processes in a busy fashion.

# Run 4 jobs at a time, starting a new job whenever one completes.
parallel -j 4 ffmpeg -r 25 -i pics/pic_%04d.png \
         -vcode libx264 -crf {1} -vpre {2} \
         anim_crf{1}_{2}.mp4 ::: 10 15 20 23 25 30 35 ::: medium slow veryslow

{1} is replaced by a choice from the first input source (following the first :::), {2} by a choice form the second input source. An example explains it more simply:

$ parallel echo {1} {2} ::: a b c :: 1 2
a 2
a 1
b 1
b 2
c 1
c 2

---

A messy approach in bash 4.3 or later (when wait -n was introduced):

max_jobs=4
running=()
for c in 10 15 20 23 25 30 35; do
  for v in medium slow veryslow; do
    ffmpg ... &
    running+=($!)
    # If we're at capacity, wait for a job to complete
    if (( ${#running[@]} == $max_jobs )); then
      wait -n
      # Any where from 0 to max_jobs - 1 could still be running
      running=( $(jobs -p) )
    fi
  done
done