Submitting jobs with numeric arguments to QSUB using Python

Question

I want to run a python function,say my_fun(x1,x2), on different nodes of cluster with SGE (QSUB). I created a script my_script.py which takes in the numeric arguments from the command line, so when running locally, I'd call it as

python my_script.py x1 x2

Now I want to submit this script to the cluster in a loop with different values of x1 and x2. Also, for the node to have access to python & the installed modules, I need to run module load Python/2.7 on the node before calling the python script through QSUB.

This seems like a pretty simple and typical use-case, but I can't find any straightforward way to do this from Python. Going back and forth between BASH and Python seems a bit clunky.

Answer 1

This more or less does what I'm looking for:

https://gist.github.com/timflutre/a9085660271bd059f71c

import sys
import subprocess

job_param1 = 12.5
job_param2 = 5.0
jobName = "python my_script.py %f %f" % (job_param1,job_param2)
cmd = "module load Python/2.7; sleep 0.2; %s" % jobName
echoArgs = ["echo", "-e", "'%s'" % cmd]
print(" ".join(echoArgs))
qsubArgs = ["qsub","-cwd"]
print(" ".join(qsubArgs))

wholeCmd = " ".join(echoArgs) + " | " + " ".join(qsubArgs)
out = subprocess.Popen(wholeCmd, shell=True, stdout=subprocess.PIPE)
out = out.communicate()[0]

jobId = out.split()[2]
print jobId