Search code examples
bashgnu-parallelgnome-terminal

How to use parallel with curl?


How do i use gnu parallel to make this process faster ?

#!/bin/bash
for (( c=1; c<=100; c++ ))
do  
    curl -sS 'https://example.com' \
        --data 'value='$c'' /dev/null
    echo $c
done

Solution

  • You can use parallel, or xargs

    seq 100 | parallel curl -sS 'https://example.com' --data value='{}' /dev/null
    seq 100 | xargs -I{} curl -sS 'https://example.com' --data value='{}' /dev/null
    

    As the script stand, output will be sent to stdout. With xargs, this will result in output from different calls potentially mixed. Consider redirect output to files for additional processing, if needed.

    You can add options for max parallel (-Pn, etc.) as needed

    I'm not sure why '/dev/null' is needed. Consider reordering:

    curl -sS --data value='{}' https://example.com'