I want to pass a file containing a list of commands to gnu-parallel while using replacement strings (e.g.: {%}).
Unfortunately, if replacement strings are used, gnu-parallel interprets the commands in the file as arguments to /bin/bash.
Here's what I want to do:
parallel -j 8 'CUDA_VISIBLE_DEVICES=$(({%} - 1)) {}' < commands.txt
where the content of commands.txt is:
/path/to/binary -arg1 a -arg2 1.0
/path/to/binary -arg1 a -arg2 1.1
...
/path/to/binary -arg1 z -arg2 9.9
However, this raises the following error:
/bin/bash: /path/to/binary -arg1 a -arg2 1.0: command not found
I was hoping GNU Parallel to run:
CUDA_VISIBLE_DEVICES=0 /path/to/binary -arg1 a -arg2 1.0
The purpose of the environment variable CUDA_VISIBLE_DEVICES is to make each process run on a different GPU (by default all processes run on the same GPU). If I didn't need CUDA_VISIBLE_DEVICES, the following code would work perfectly:
parallel -j 8 < commands.txt
How can I get around this?
While --colsep may work sometimes, it is not always the correct choice. This will create the files abc and def:
echo 'touch abc\ def' | parallel -v --colsep ' ' A=B {}
Normally it will be better to de-quote the expression using eval:
echo 'touch abc\ def' | parallel -v eval A=B {}
So:
parallel -j 8 'eval CUDA_VISIBLE_DEVICES=$(({%} - 1)) {}' < commands.txt
If you use $(({%} - 1)) a lot, consider making your own replacement string:
echo '--rpl {%-1}\ $_=slot()-1' >> ~/.parallel/config
parallel -j 8 'eval CUDA_VISIBLE_DEVICES={%-1} {}' < commands.txt
Or even:
echo '--rpl '"'"'{CUDA} $_="CUDA_VISIBLE_DEVICES=".(slot()-1)'"'" >> .parallel/config
parallel -j 8 'eval {CUDA} {}' < commands.txt