bashparallel-processing

How to run given function in Bash in parallel?


There have been some similar questions, but my problem is not "run several programs in parallel" - which can be trivially done with parallel or xargs.

I need to parallelize Bash functions.

Let's imagine code like this:

for i in "${list[@]}"
do
    for j in "${other[@]}"
    do
    # some processing in here - 20-30 lines of almost pure bash
    done
done

Some of the processing requires calls to external programs.

I'd like to run some (4-10) tasks, each running for different $i. Total number of elements in $list is > 500.

I know I can put the whole for j ... done loop in external script, and just call this program in parallel, but is it possible to do without splitting the functionality between two separate programs?


Solution

  • Edit: Please consider Ole's answer instead.

    Instead of a separate script, you can put your code in a separate bash function. You can then export it, and run it via xargs:

    #!/bin/bash
    dowork() { 
        sleep $((RANDOM % 10 + 1))
        echo "Processing i=$1, j=$2"
    }
    export -f dowork
    
    for i in "${list[@]}"
    do
        for j in "${other[@]}"
        do
            printf "%s\0%s\0" "$i" "$j"
        done
    done | xargs -0 -n 2 -P 4 bash -c 'dowork "$@"' --