If I build CPython from source on Windows I encounter problems when I want to pip install a package that contains a C-Extension. It seems like the error happens while linking the libraries.
For example when installing cython (but it also crashes with the same error on other C extension packages):
LINK : fatal error LNK1104: cannot open file 'python38.lib'
error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.23.28105\bin\HostX86\x86\link.exe' failed with exit status 1104
The reason why it cannot open the "python38.lib" is because the ".lib" file in debug mode is called "python38_d.lib".
A minimal reproducible example would be (on the command-line) based on the Quick Reference of the CPython developer guide:
git clone --branch v3.8.0 https://github.com/python/cpython.git
cd cpython
git checkout v3.8.0
.\PCbuild\build.bat -e -d
.\PCbuild\win32\python_d.exe -m ensurepip
.\PCbuild\win32\python_d.exe -m pip install pip --upgrade -vv
.\PCbuild\win32\python_d.exe -m pip install setuptools --upgrade -vv
.\PCbuild\win32\python_d.exe -m pip install cython -vv
The resulting distutils.sysconfig.get_config_vars() is:
{'BINDIR': '...\\cpython\\PCbuild\\win32',
'BINLIBDEST': ...\\cpython\\Lib',
'EXE': '.exe',
'EXT_SUFFIX': '_d.cp38-win32.pyd',
'INCLUDEPY': '...\\cpython\\include;...\\cpython\\PC',
'LIBDEST': '...\\cpython\\Lib',
'SO': '_d.cp38-win32.pyd',
'VERSION': '38',
'exec_prefix': '...\\cpython',
'prefix': '...\\cpython',
'srcdir': '...\\cpython'}
Is there something I'm missing? Is building C-Extensions on Python-debug builds on Windows simply not supported? If it is supported: how would I do it?
Linking against pythonXY.lib is a little bit sneaky on Windows. When you look at the command line for linking, you will see that no python-library is passed to the linker, i.e. 'link.exe`. Note: This is also the case for Linux, but on Linux one doesn't have to because the needed symbols will be provided by the python-executable.
However, it is easy to check via dumpbin /dependents resulting.pyd, that there is a dependency on pythonXY.dll, also adding extra_link_args = ["/VERBOSE:LIB"] to extension-definition and triggering verbose-mode of the linker will show that the linker uses pythonXY.lib.
The sneaky part: Microsoft Compler has a convinience-pragma #pragma comment(lib, ...) to automatically trigger linking of a library, which is also used in Python-headers:
# if defined(_MSC_VER)
/* So MSVC users need not specify the .lib
file in their Makefile (other compilers are
generally taken care of by distutils.) */
# if defined(_DEBUG)
# pragma comment(lib,"python39_d.lib")
# elif defined(Py_LIMITED_API)
# pragma comment(lib,"python3.lib")
# else
# pragma comment(lib,"python39.lib")
# endif /* _DEBUG */
# endif /* _MSC_VER */
As you can see, to link against the debug version, one needs to define _DEBUG.
_DEBUG is automatically defined by distutils on Windows, if build_ext is called with options --debug, e.g.
python setup.py build_ext -i --debug
That can be translated to pip as
pip install --global-option build --global-option --debug XXXXX
which can be interpreted roughly as: trigger build command (which also includes build_ext-command) with option --debug prior to installing.
Another subtility when building debug C-extensions, there is more to it on Windows:
#ifdef _DEBUG
# define Py_DEBUG
#endif
Having defined Py_DEBUG macro meant incompartible ABIs until Python3.8, because it also assumed Py_TRACE_REFS which leads to different memory layout of PyObject and some additional functionality missing in the release-mode.
However, since Python3.8, one probably can get away with it by providing the missing pythonXY_d.lib/pythonYX.lib as a symlink linking to another version.