Search code examples
bashcurlgnu-parallel

How to run curl commands parallel using GNU parallel


Just recently started programming in bash. I want to run curl command for 'n' times where n is user input. And I want to run in parallel. I came across GNU Parallel. the curl code for running n times is

ip1="some ip"
ip2="some ip"
ip3="some ip"
for ((j=1; j<= $servers;))
do
    for ((i= 1; i<= $val; i++))
    do
        c=ip$j
        value=scatest"$j$i"
        curl -s -X POST ${!c}:4000/invoke -H "content-type: application/json" -d '{"args":["org1","'$value'","27-06-2018"]}'
        if ((i % $val == 0))
        then
            ((j++))
        fi
    done
done

i want to run this curl command equally and parallel on these different hosts based on the user input. Can any one help me out?

Thanks in advance


Solution

  • It is fairly simple to parallelize per server:

    val=1000
    doit() {
        server="$1"
        serverno="$2"
        for ((i= 1; i<= $val; i++))
        do
            value=scatest"$serverno$i"
            curl -s -X POST ${server}:4000/invoke -H "content-type: application/json" -d '{"args":["org1","'$value'","27-06-2018"]}'
        done
    }
    export -f doit
    export val
    parallel -j0 doit {} {#} ::: serverip1 serverip2 ...
    

    If you can accept scatest has a different value you can parallelize even further:

    doit() {
        server="$1"
        jobno="$2"
        i="$3"
        value=scatest"$jobno$i"
        curl -s -X POST ${server}:4000/invoke -H "content-type: application/json" -d '{"args":["org1","'$value'","27-06-2018"]}'
    }
    export -f doit
    seq $val | parallel -j0 doit {1} {#} {2} ::: serverip1 serverip2 ... :::: -