Search code examples
bashgnu-parallel

Process file contents with gnu-parallel

I have a file with contents as follows:

/path/to/file1
/path/to/file2
/path/to/file3
/path/to/file4

I want to run a command on each line in parallel using the gnu-parallel utility which I'm aware supports file inputs using ::::. What I'm unsure about is that which arguments should I pass to gnu-parallel to split the file contents by \n and process in parallel?

Solution

  • Here's man parallel:

    NAME
       parallel - build and execute shell command lines from
       standard input in parallel

SYNOPSIS
       parallel [options] [command [arguments]] < list_of_arguments
       [...]

    Here's an example of this form of invocation:

    $ cat mylist.txt
/path/to/file1
/path/to/file2
/path/to/file3
/path/to/file4

$ parallel 'echo "I am processing:" {}' < mylist.txt
I am processing: /path/to/file1
I am processing: /path/to/file2
I am processing: /path/to/file3
I am processing: /path/to/file4