I am trying to assign tasks to qsub from a python script.
So far I have always been using such expressions for qsub submission:
echo "python script.py arg1 arg2" | qsub
I tried to reproduce this command in python:
command = 'echo "python sub_master_script.py ' + str(i) + ' ' + path + '" | qsub -pe make 5 -N Teaching_' + str(i) + ' -cwd'
subprocess.call(shlex.split(command))
Although command carries a command I would normally use, it is interpreted as a simple echo when I launch the python script.
And instead of starting a new job, I get this written in console:
python sub_master_script.py 0.75 /data5/bio/runs-galkin/Iterative/test_OTU.txt | qsub -pe make 5 -N Teaching_0.75 -cwd
Why this happens? How can I make it work?
you're making heavy use of shell features, mostly by running 2 process piped together.
subprocess.call(shlex.split(command),shell=True)
is a quickfix.
A proper way of doing it would be to use 1 subprocess.Popen instance, provide the input through communicate (to simulate echo python ...) and drop shell=True (using pipes.quote as Charles commented to make sure that the strings are properly quoted if needed)
import pipes
input_string = "python sub_master_script.py {} {}\n".format(*(map(pipes.quote,[str(i),path])))
p = subprocess.Popen(['qsub','-pe','make','5','-N','Teaching_{}'.format(i),'-cwd'],stdin=subprocess.PIPE)
out,err = p.communicate(input_string)