Search code examples
gnu-parallel

GNU Parallel: How to recieve stdin from gnu parallel pipe as if it were from a file?

Suppose I have the following command which runs like so: 

my_command -f file

How do I set up parallel such that content piped in is executed as if it were a pipe?

# For example, in pseudo code: 

cat item | parallel my_command -f <(the incoming stdin)

It is important to do it this way, as I am processing specific chunks of data via the --speadstdin option, so if there as a way to redirect the piped input as if it were coming from a file, that would solve for the situation.

Solution

  • You are looking for --cat or --fifo:

    cat file | parallel --pipe --cat my_command -f {}

    Faster, but requires my_command read the complete file from start to end once and only once:

    cat file | parallel --pipe --fifo my_command -f {}

    Faster, but requires file is a real file:

    parallel -a file --pipepart --block -1 --fifo my_command -f {}

    See also chapter 9.6 in GNU Parallel 2018 (Online: https://doi.org/10.5281/zenodo.1146014 Printed: http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html).