Search code examples
multithreadingshellloopswgetjobs

How to control the number of background job in shell script

I have a shell script to download a batch of files and want to controlling it to have certain number of background download jobs.

Solution 
  • json='list.json'
wget -o $json http://fileserver/getFileList

jobsCount=0
while IFS= read -r url ; do

        #download the file
        wget -q $url &

        #contorling the amount of concurrent jobs
          while true; do
            #get current background job count
            jobsCount=$(jobs -p | wc -l)

            if [ $jobsCount -lt 6 ]; then
              echo "jobsCount $jobsCount"
              break
            fi
            echo "jobsCount $jobsCount (Sleep)"
            sleep 1
          done

done < <(jq -r '.data[] | (.URL)' <$json)
#wait for last batch of job to finish
wait