Search code examples
bashunixparallel-processinggnu-parallel

Parallelizing jobs execution in bash

I am trying to parallelize execution of some commands with gnu-parallel

cmd_arr=()
  for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
        cmd_arr+=("aomenc $x.y4m --i444 --enable-qm=1 --qm-min=$i --profile=3 -w $width -h $height -b 10 --end-usage=q --cq-level=$q -o ${x}_${q}_${i}.ivf")
    done
  done
  parallel --will-cite  ::: "${cmd_arr[@]}"
  cmd_arr=()
  for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
        cmd_arr+=("aomdec ${x}_${q}_${i}.ivf --output-bit-depth=10 -o ${x}_${q}_${i}.y4m")
    done
  done
  parallel --will-cite  ::: "${cmd_arr[@]}"
  cmd_arr=()
  for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
        cmd_arr+=("ffmpeg -nostats -loglevel 0 -y -i ${x}_${q}_${i}.y4m ${x}_${q}_${i}.png")
    done
  done
  parallel --will-cite  ::: "${cmd_arr[@]}"
  cmd_arr=()
  for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
        cmd_arr+=("convert ${x}_${q}_${i}.png PNG24:${x}_${q}_${i}_ssimulacra.png")
    done
  done
  parallel --will-cite  ::: "${cmd_arr[@]}"

When i execute the code above , convert seems to failed stating that it can't find the png to convert.

Execution should happen in that order aomenc aomdec ffmpeg convert

It works perfectly fine serially

for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
    aomenc "$x".y4m --i444 --enable-qm=1 --qm-min="$i" --profile=3 -w "$width" -h "$height" -b 10 --end-usage=q --cq-level="$q" -o "$x"_"$q"_"$i".ivf
        aomdec "$x"_"$q"_"$i".ivf --output-bit-depth=10 -o "$x"_"$q"_"$i".y4m
        ffmpeg -nostats -loglevel 0 -y -i "$x"_"$q"_"$i".y4m "$x"_"$q"_"$i".png
        convert "$x"_"$q"_"$i".png PNG24:"$x"_"$q"_"$i"_ssimulacra.png 
done

Here is the output , i get the issue only for some files which make me belief the bug has something to do with parallel

convert: unable to open image `bench.png_20_4.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_20_4.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_20_4_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_20_8.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_20_8.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_20_8_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_20_12.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_20_12.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_20_12_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_22_4.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_22_4.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_22_4_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_22_8.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_22_8.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_22_8_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_22_10.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_22_10.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_22_10_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_22_12.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_22_12.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_22_12_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_24_4.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_24_4.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_24_4_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_24_6.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_24_6.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_24_6_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_24_8.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_24_8.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_24_8_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_24_12.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_24_12.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_24_12_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_26_6.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_26_6.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_26_6_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_26_10.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_26_10.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_26_10_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.

The y4m file exists , i confirmed it by running the command manually for the affected file.Why isn't ffmpeg converting all y4m to png ?

Solution

  • If your serial code works then this ought to, too:

    #!/bin/bash

doit() {
    x="$1"
    q="$2"
    i="$3"
    aomenc "$x".y4m --i444 --enable-qm=1 --qm-min="$i" --profile=3 -w "$width" -h "$height" -b 10 --end-usage=q --cq-level="$q" -o "$x"_"$q"_"$i".ivf
    aomdec "$x"_"$q"_"$i".ivf --output-bit-depth=10 -o "$x"_"$q"_"$i".y4m
    ffmpeg -nostats -loglevel 0 -y -i "$x"_"$q"_"$i".y4m "$x"_"$q"_"$i".png
    convert "$x"_"$q"_"$i".png PNG24:"$x"_"$q"_"$i"_ssimulacra.png 
}
export -f doit

parallel doit {1} {2} {3} ::: "$x" ::: {10..40..2} ::: {4..12..2}

    You can test by hand on a single combination before using GNU Parallel:

    doit "$x" 10 12

    Building an array of commands is typically not needed.

    If you have local (non-exported) variables that you need in the function, you can use env_parallel instead of parallel:

    #!/bin/bash

. `which env_parallel.bash`

# Do this in a clean environment without any variables set
# It only needs to be run once, so it does not have to be part of the script
env_parallel --record-env

# Set the local variables after doing --record-env
myvar=...

doit() {
    x="$1"
    q="$2"
    i="$3"
    echo "$myvar"
    aomenc "$x".y4m --i444 --enable-qm=1 --qm-min="$i" --profile=3 -w "$width" -h "$height" -b 10 --end-usage=q --cq-level="$q" -o "$x"_"$q"_"$i".ivf
    aomdec "$x"_"$q"_"$i".ivf --output-bit-depth=10 -o "$x"_"$q"_"$i".y4m
    ffmpeg -nostats -loglevel 0 -y -i "$x"_"$q"_"$i".y4m "$x"_"$q"_"$i".png
    convert "$x"_"$q"_"$i".png PNG24:"$x"_"$q"_"$i"_ssimulacra.png 
}

# --env _ = ignore variables defined before --record-env
env_parallel --env _ doit {1} {2} {3} ::: "$x" ::: {10..40..2} ::: {4..12..2}

    Consider spending an hour walking through https://www.gnu.org/software/parallel/parallel_tutorial.html Your command line will love you for it.