Search code examples
pythongccsetuptoolsdistutils

Distutils partially ignoring CC environment variable when building python extension

I am trying to install the subprocess32 python module (https://github.com/google/python-subprocess32) and am having some issues with distutils. The module includes a C extension that has to be built, but when I run either pip install . or python setup.py install I get the following output:

...
creating build
creating build/temp.linux-x86_64-2.7
/non/existent/path/gcc -pthread -fPIC ... -c _posixsubprocess.c -o build/temp.linux-x86_64-2.7/_posixsubprocess.o
unable to execute '/non/existent/path/gcc': No such file or directory

Obviously distutils is using the wrong path to gcc for some reason. I then tried to manually specify the right path to gcc using export CC=/correct/path/to/gcc and I get this output:

building '_posixsubprocess' extension
creating build/temp.linux-x86_64-2.7
/correct/path/to/gcc -fPIC -fno-strict-aliasing -g -O2 ... -c _posixsubprocess.c -o build/temp.linux-x86_64-2.7/_posixsubprocess.o
/non/existent/path/gcc -pthread -shared ... build/temp.linux-x86_64-2.7/_posixsubprocess.o -o build/lib.linux-x86_64-2.7/_posixsubprocess.so
unable to execute '/non/existent/path/gcc': No such file or directory

The original problematic command is now using the correct path, but it is still trying to use the incorrect location of gcc to build the shared library. Is there another environment variable I need to specify to correct this behaviour?

Solution

  • I've experimented the exactly same problem with you. I spent some time digging into distutils source code and found the problem.

    distutils will use the link config from when Python is built. In this case, the gcc used for building python is different with the gcc you're using to build extension.

    Run this command too see the default link command:

    python2 "from distutils.sysconfig import get_config_var; print(get_config_var('LDSHARED'))"

    You should found the incorrent gcc is in the start of the config, for example:

    /non/existent/path/gcc -pthread -shared

    First solution

    Set LDSHARED enviroment variable to overwrite it with the corrent gcc path:

    export LDSHARED="/correct/path/to/gcc -pthread -shared"

    And then rebuild the extension, it should works.

    Second solution (and possibly better)

    The LDSHARED config is retrieved from lib/python2.7/_sysconfigdata.py file, which is generated at build time.

    You can modify this file so there's no need to set environment variables.

    Debug skills

    Set DISTUTILS_DEBUG environment to activate debug mode, so you can see the traceback when compilation fails.