Search code examples
linuxparallel-processingxargsgnu-parallel

Parallel execution of a job which takes multiple inputs on Linux

I have a file with a list of paths. e.g.:

/run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/CNMC_N_191/ /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/204-196/

I have a program (let's call it 'prog') which takes 3 parameters, 2 input filenames and 1 output filename. Normal execution:

prog ~/A.txt ~/B.txt ~/out.txt

I want to execute in parallel my program with parameters constructed based on each line of the file, e.g.:

prog /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/CNMC_N_191/A.txt /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/CNMC_N_191/B.txt /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/CNMC_N_191/out.txt
prog /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/204-196/A.txt /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/204-196/B.txt /run/user/1007/gvfs/smb-share:server=xeonator,share=c/Results/Normals/8_mo/204-196/out.txt

I have been trying to use xargs and GNU parallel, but I can't construct a proper command line for those tools. Here is my attempt:

cat ListPaths.txt | xargs -I '{}' -n 1 -P 8 ./SkullSegmenter/SkullSegmenter '{}'A.txt '{}'B.txt '{}'out.txt

But this does not construct the arguments properly. Can someone help me with this?

Solution

  • This should do it:

    cat ListPaths.txt | parallel ./SkullSegmenter/SkullSegmenter {}A.txt {}B.txt {}out.txt

    Test with --dry-run to see if these are the commands you want run:

    cat ListPaths.txt | parallel --dry-run ./SkullSegmenter/SkullSegmenter {}A.txt {}B.txt {}out.txt