Search code examples
bashcurlparallel-processingshgnu-parallel

How to curl in parallel from a list of links


list.txt contains links. I need them to be curled in parallel at the same time. Meaning curling example.com, abc.com and 123.com at the same time.

What I've tried

mycurl() {
curl $1
}
seq 1 | parallel -j0 mycurl list.txt >> output.txt
#I've also tried
seq 1 | parallel -j0 mycurl ::: list.txt >> output.txt
#I"ve also tried
parallel -a list.txt mycurl >> output.txt

They all output parallel --help

list.txt contents

example.com
abc.com
def.com

Solution

  • I fail to see the point of your mycurl() function, but you can use it like this if you want to:

    # Declare 'mycurl()' function
    mycurl() {
       # list.txt can be accessed in here if you want
       curl $1
    }
    
    # Make 'mycurl()' function available to processes started by 'parallel'
    export -f mycurl
    
    # Call 'mycurl()' once with each line of 'list.txt' as parameter
    parallel -a list.txt mycurl {} > output.txt