Run bash command in parallel for N-processes in batch

Question

I have a command:

python test.py -i input_file -p parameter_1 -o output_file

I want to run this for multiple input files, 4 input files at a time (in parallel) and once they finish run the next 4 and so on using a bash script.

I am not able to find the right answer.

I am trying the following but I am not sure if it is right

for (( n=0; n<12; n++ ))
do
    ((j=j%4)); ((j++==0)) && wait
    python test.py -i input_list[n] -p parameter_1 -o output_$n &
done

Thanks in advance!

Answer 1

In a bash script, you can try something like:

processes_max=4
counter=0

for f in input_files/*.txt
do
    python test.py -i $f -p parameter_1 -o output_files/$(basename $f) &  
    counter=$((counter+1))
    # if the counter equal to processes_max: wait for all processes to finish
    if [ $counter -eq $processes_max ]; then
        wait
        counter=0
    fi
done

wait