Bash: Logging stdout from multiple xargs parallel processes to separate log files

Question

I am processing a text file with multiple parallel processes spawned by xargs. I also need to capture the stdout from each process into a separate log file. Below is an example where the output from each process is interleaved into a single file -- not what I want.

Ideally, each logfile should be numbered by the file line number, that is, logfile-1, logfile-2, etc.

cat inputfile.txt | xargs -n 1 -P 8 ./myScript.sh | tee logfile

It would be nice to avoid an external wrapper script if possible, but if there is a way to wrap myScript with a here document, that would work.

Answer 1

Try this:

nl inputfile.txt | xargs -n 2 -P 8 sh -c './myScript.sh "$1" > logfile-$0'

This assumes each argument in inputfile.txt is on its own line and contains no spaces. The nl command numbers each line, which pairs each argument with a unique number. The xargs commands takes two arguments at time, the first the line number, the second the corresponding line from inputfile.txt, and passes them to sh. The sh command uses the arguments to generate the output file name and the argument to myScript.sh respectively.