Forewarning, question is probably more about a lack of understanding of the bsub command and login shells than python's Popen().
I am trying to submit a LSF script within a python script using subprocess.Popen()
pipe = subprocess.Popen(shlex.split("bsub < script.lsf"),
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
It seems a subprocess is properly launched and the command executed, but I get an error in stderr from the bsub command that reads "Empty job. Job not submitted."
I was worried it had something to do with subprocess launching a login-shell, so I tried to use shell=True within the Popen command to alleviate that.
pipe = subprocess.Popen("bsub < script.lsf", shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
This successfully submits the job as expected. So my question is how can I submit this job without using shell=True?
I've tried using some bsub options such as
pipe = subprocess.Popen(shlex.split("bsub -L /bin/sh < script.lsf"),
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
with the same "Empty job. Job not submitted." error returned. I feel this is close to what I am looking for, but I don't fully understanding what "The name of the login shell (must specify an absolute path" is exactly. Perhaps it is not /bin/sh on the system I am using. Is there a way to print this path?
< and > are not arguments to your command (in this case bsub), but are instructions to the shell about where stdin or stdout (respectively) should be directed before that command is started.
Thus, the appropriate replacement is to specify this redirection with a separate argument to Popen:
pipe = subprocess.Popen(['bsub', '-L', '/bin/sh'],
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
stdin=open('script.lsf', 'r'))