Using subprocess for compiling a c++ program from python script

Question

I know there are some similar questions, here Invoking C compiler using Python subprocess command and subprocess, invoke C-program from within Python but I believe my question is in some sense different.

I need to compile a c++ program which uses some ROOT libraries so I need to add some flags and link some libraries for the compilation. Therefore my compilation line on the normal shell is:

> $($ROOTSYS/bin/root-config --cxx) $($ROOTSYS/bin/root-config --cflags --glibs) Analysis.cxx -o analysis.exe

which works nicely. I want to do this compilation from my python script. I have read the documentation for the subprocess module but I could not get a solution without using shell=True in the call of subprocess.Popen and I do not really undestand the difference. If I use:

process = Popen(["$($ROOTSYS/bin/root-config --cxx) $($ROOTSYS/bin/root-config --cflags --glibs) Analysis.cxx -o analysis.exe"], shell=True)

does the job. However, this:

process = Popen(["$($ROOTSYS/bin/root-config --cxx)", "$($ROOTSYS/bin/root-config --cflags --glibs)", "Analysis.cxx", "-o", "analysis.exe"])

I got the following:

    Traceback (most recent call last):
  File "make_posanalysis.py", line 45, in <module>
    "Analysis.cxx", "-o", "analysis.exe"])
  File "Python/2.7.15/x86_64-slc6-gcc62-opt/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "Python/2.7.15/x86_64-slc6-gcc62-opt/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

I would like to understand the difference between using/not using shell=True since it seems to be the reason behind making the script work or not. Or, is there something else I am missing?

Answer 1

From the documentation:

If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:

Popen(['/bin/sh', '-c', args[0], args[1], ...])

So it'e executing something equivalent to:

/bin/sh -c '$($ROOTSYS/bin/root-config --cxx)' '$($ROOTSYS/bin/root-config --cflags --glibs)' "Analysis.cxx", "-o", "analysis.exe"

This isn't what you want, because it only performs $(...) expansion in the first argument; everything else is taken literally, and become the positional arguments if the command in the first argument refers to $1, $2, etc.

If you want everything parsed by the shell, just give a single string.