Search code examples
curlxargs

Get xargs sequence number

In GNU Parallel you have:

seq()    -     sequence number of job

Thanks to this I can run wget in parallel using:

parallel -a urls.txt -j8 wget -q {} -O '{=$_=sprintf("file%05d.ts", $job->seq())=}'

..and it will name files like so:

file00001.ts
file00002.ts
...
file00999.ts

Now, how can I get the sequence number of xargs?

Example:

xargs -P 8 -n 1 curl -O < urls.txt
Solution

  • Like this:

    cat urls.txt |
  parallel --shellquote |
  parallel --shellquote |
  perl -pe 's/(.*)/$1.sprintf(" -O file%05d.ts",++$t)/e' |
  xargs -P 8 -n 1 -I {} bash -c "wget {}"

    The 2 --shellquotes are needed if you have urls with special chars like space and ':

    http://pi.dk/? '