Parallel python and subprocess

Question

I'm trying to make a cluster using Parallel Python (pp) module. I have set up the cluster. Nodes are available:

./ppserver.py -p 35000 -i 127.0.0.1 -s "mysecret"

master is also configured and jobs are submitted:

ppservers=("node-1:35000", "node-2:35000")
job_server = pp.Server(ppservers=ppservers, secret="mysecret")

f1 = job_server.submit(SomeFunction, argus1), modules=('subprocess',))
f2 = job_server.submit(SomeFunction, argus2), modules=('subprocess',))

a = f2()
b = f1()

SomeFunction:

def SomeFunction(argus):
  proc = subprocess.check_output(['python',
                    '~/path/to/python_script.py',
                    '--argu1', argus[0],
                    '--argu2', argus[1],
    ])
  return proc

python_script.py talks to youtube api and writes data into mysql database.

on execution i'm getting this error:

An error has occured during the function execution
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/ppworker.py", line 90, in run
    __result = __f(*__args)
  File "<string>", line 16, in SomeFunction
  File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['python', '~/path/to/python_script.py', '--argu1', 'argu1', '--argu2', 'argu2']' returned non-zero exit status 2

python_script.py is fully debuged, any idea?

Answer 1

That's because your path contains a ~, which isn't expanded by default.

2 ways to fix that:

• add shell=True argument to run through a shell (not recommended)
• best way: use os.path.expanduser

like this:

proc = subprocess.check_output(['python',
                    os.path.expanduser('~/path/to/python_script.py'),
                    '--argu1', 'argu1',
                    '--argu2', 'argu2',