subprocess.Popen() won't run python script, python command not found

Question

I'm trying to get subprocess.Popen to run a python script but I keep getting the following error: /bin/sh: python: command not found. The script takes a yaml file as an argument. I've tried this line both with and without shell=True. The script runs fine when I run it with the python command in my Linux terminal. What am I doing wrong?

    process = subprocess.Popen(
        ['python', PATH_TO_PYTHON_SCRIPT, PATH_TO_CONFIG_FILE],
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT, stdin=subprocess.PIPE, shell=True)
    with process.stdout, open(processing_log, 'ab') as f_in:
        for line in iter(process.stdout.readline, b''):
            f_in.write(line)

Answer 1

If you want to run with the same interpreter you're currently running in, I'd suggest passing sys.executable rather than 'python' so you're not dependent on the vagaries of PATH lookup; if you want to look it up from the PATH, you might try using shutil.which to look it up at the Python layer to minimize the number of things in the way.

Side-note: Why are you running this with shell=True? That adds a number of complicating factors that you should probably avoid, and you're not taking advantage of the tiny benefits it provides anyway.

Additionally, if all you want to do is append the output to a given file, you can always just do:

with open(processing_log, 'ab') as f_out:
    process = subprocess.Popen(
        ['python', PATH_TO_PYTHON_SCRIPT, PATH_TO_CONFIG_FILE],
        stdout=f_out,
        stderr=subprocess.STDOUT, stdin=subprocess.PIPE)

and let the process write to the file directly instead of acting as a passthrough layer.