Building Python C module on Windows
I am trying to build a 'C' python extension on Windows, the core C code compiles absolutely fine, but I am unable to build the python module using setuptool as I am getting
mandlebrot.c(36): fatal error C1083: Cannot open include file: 'stdio.h': No such file or directory
error: command 'e:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.35.32215\bin\HostX86\x86\cl.exe' failed with exit code 2
I am trying to build using:
python3.9 setup.py build_ext --inplace --plat-name win32
Any help would be welcome as I have spent a long time trying to work out how to get this to work.
The setup file:
from setuptools import setup, Extension
from subprocess import getoutput
LIB_DIRS=['.\mandlebrot_set\libs']
INCLUDE_DIRS=['.\mandlebrot_set\includes']
module1 = Extension("mandlebrot",
sources = ["mandlebrot.c", "mandlebrot_python.c"],
libraries = ['mpfr', 'mpir'],
library_dirs = LIB_DIRS,
include_dirs = INCLUDE_DIRS)
setup(name = "PackageName",
version = '0.1',
description = 'Mandlebrot Set calculator',
author = '',
author_email = '',
ext_modules = [module1]
)
1. The error
By default, Python is built on Win with VStudio ([Python.Wiki]: WindowsCompilers), and it also uses that to build C / C++ code (unless otherwise instructed).
It seems like you don't have VStudio's cross build tools (for 032bit) installed. A quick search revealed:
My installation (Visual Studio Installer) looks like:
Make sure you have installed:
The build tools
At least one Windows SDK version
Try building a simple Hello World application ([MS.DevBlogs]: C++ Tutorial: Hello World) for 032bit (Win32)
Some VStudio links that might help:
Once the build works, try again the extension
If any of the 2 previous steps still fails, you might think of repairing (or even reinstalling - as a last resort) your VStudio installation
2. Setup
Since there are files not provided for a MCVE ([SO]: How to create a Minimal, Reproducible Example (reprex (mcve))), I'm going to use some from [SO]: How to create python C++ extension with submodule that can be imported (@CristiFati's answer) (latest one that I worked on involving extension modules).
I also want to point out setup.py deprecation ([SO]: 'setup.py install is deprecated' warning shows up every time I open a terminal in VSCode), but I'm not going to insist on that.
setup.py:
#!/usr/bin env python
import os
from setuptools import Extension, setup
# @TODO - cfati: Use sources from the other SO question
SOURCE_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "q076222409")
# Define the extension module
extension_mod = Extension("parent",
sources=[os.path.join(SOURCE_DIR, "custom.cc")])
# Define the setup parameters
setup(
name="parent",
version="1.0",
description="A C++ extension module for Python.",
ext_modules=[extension_mod],
)
I have 2 separate dirs for 064bit (pc064) and 032bit (pc032) builds (1) (you'll see later why).
3. Build for pc064
This is native (or direct) build, as (CPU and) default Python installation (that I'm going to use) is pc064.
Output:
[cfati@CFATI-5510-0:e:\Work\Dev\StackExchange\StackOverflow\q076249589\pc064]> sopr.bat ### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ### [prompt]> [prompt]> dir /b [prompt]> [prompt]> dir /b .. pc032 pc064 setup.py [prompt]> [prompt]> :: Build for pc064 [prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts\python.exe" ..\setup.py build_ext --inplace running build_ext building 'parent' extension creating build creating build\temp.win-amd64-cpython-310 creating build\temp.win-amd64-cpython-310\Release creating build\temp.win-amd64-cpython-310\Release\Work creating build\temp.win-amd64-cpython-310\Release\Work\Dev creating build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange creating build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow creating build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409 C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\bin\HostX86\x64\cl.exe /c /nologo /O2 /W3 /GL /DNDEBUG /MD -Ie:\Work\Dev\VEnvs\py_pc064_03.10_test0\include -Ic:\Install\pc064\Python\Python\03.10\include -Ic:\Install\pc064\Python\Python\03.10\Include -IC:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\include -IC:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\ATLMFC\include -IC:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Auxiliary\VS\include "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /EHsc /Tpe:\Work\Dev\StackExchange\StackOverflow\q076 222409\custom.cc /Fobuild\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\custom.obj custom.cc creating e:\Work\Dev\StackExchange\StackOverflow\q076249589\pc064\build\lib.win-amd64-cpython-310 C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:e:\Work\Dev\VEnvs\py_pc064_03.10_test0\libs /LIBPATH:c:\Install\pc064\Python\Python\03.10\libs /LIBPATH:c:\Install\pc064\Python\Python\03.10 /LIBPATH:e:\Work\Dev\VEnvs\py_pc064_03.10_test0\PCbuild\amd64 /LIBPATH:C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\ATLMFC\lib\x64 /LIBPATH:C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\lib\x64 "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22000.0\\um\x64" /EXPORT:PyInit_parent build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\custom.obj /OUT:build\lib.win-amd64-cpython-310\parent.cp310-win_amd64.pyd /IMPLIB:build\temp.win-am d64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\parent.cp310-win_amd64.lib Creating library build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\parent.cp310-win_amd64.lib and object build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\parent.cp310-win_amd64.exp Generating code Finished generating code copying build\lib.win-amd64-cpython-310\parent.cp310-win_amd64.pyd -> [prompt]> [prompt]> dir /b build parent.cp310-win_amd64.pyd [prompt]> [prompt]> :: Try importing in Python pc064 [prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts\python.exe" -c "import sys;print(sys.version);from parent.child import hello;print(hello());print(\"Done.\n\")" 3.10.9 (tags/v3.10.9:1dd9be6, Dec 6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)] Hi, World! Done.
So far, so good.
4. Build for pc032
This is cross build (as host and target architectures don't match).
According to [Python.Docs]: Creating Built Distributions - Cross-compiling on Windows (emphasis is mine):
To cross-compile, you must download the Python source code and cross-compile Python itself for the platform you are targeting - it is not possible from a binary installation of Python (as the .lib etc file for other platforms are not included).
Although it's old, and the other way around it also applies (partially) to our situation.
As explained in the (build related) URLs above, when building for Win, the linker needs the .lib files (${PYTHONCORE}.lib in our case) for the target architecture, so they are required to exist on the host, and also setup.py (SetupTools) needs to find them.
Output (I'll reuse this console):
[cfati@CFATI-5510-0:e:\Work\Dev\StackExchange\StackOverflow\q076249589\pc032]> sopr.bat ### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ### [prompt]> [prompt]> dir /b [prompt]> [prompt]> dir /b .. pc032 pc064 setup.py [prompt]> [prompt]> :: Build for pc032 [prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts\python.exe" ..\setup.py build_ext --inplace --plat-name win32 -L"e:\Work\Dev\VEnvs\py_pc032_03.10_test0\libs" running build_ext building 'parent' extension creating build creating build\temp.win-amd64-cpython-310 creating build\temp.win-amd64-cpython-310\Release creating build\temp.win-amd64-cpython-310\Release\Work creating build\temp.win-amd64-cpython-310\Release\Work\Dev creating build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange creating build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow creating build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409 C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\bin\HostX86\x86\cl.exe /c /nologo /O2 /W3 /GL /DNDEBUG /MD -Ie:\Work\Dev\VEnvs\py_pc064_03.10_test0\include -Ic:\Install\pc064\Python\Python\03.10\include -Ic:\Install\pc064\Python\Python\03.10\Include -IC:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\include -IC:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\ATLMFC\include -IC:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Auxiliary\VS\include "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /EHsc /Tpe:\Work\Dev\StackExchange\StackOverflow\q076 222409\custom.cc /Fobuild\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\custom.obj custom.cc creating e:\Work\Dev\StackExchange\StackOverflow\q076249589\pc032\build\lib.win-amd64-cpython-310 C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\bin\HostX86\x86\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:e:\Work\Dev\VEnvs\py_pc032_03.10_test0\libs /LIBPATH:e:\Work\Dev\VEnvs\py_pc064_03.10_test0\libs /LIBPATH:c:\Install\pc064\Python\Python\03.10\libs /LIBPATH:c:\Install\pc064\Python\Python\03.10 /LIBPATH:e:\Work\Dev\VEnvs\py_pc064_03.10_test0\PCbuild\win32 /LIBPATH:C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\ATLMFC\lib\x86 /LIBPATH:C:\Install\pc064\Microsoft\VisualStudioCommunity\2022\VC\Tools\MSVC\14.35.32215\lib\x86 "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.22000.0\ucrt\x86" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22000.0\\um\x86" /EXPORT:PyInit_parent build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\custom.obj /OUT:build\lib.win-amd64-cpython-310 \parent.cp310-win_amd64.pyd /IMPLIB:build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\parent.cp310-win_amd64.lib Creating library build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\parent.cp310-win_amd64.lib and object build\temp.win-amd64-cpython-310\Release\Work\Dev\StackExchange\StackOverflow\q076222409\parent.cp310-win_amd64.exp Generating code Finished generating code copying build\lib.win-amd64-cpython-310\parent.cp310-win_amd64.pyd -> [prompt]> [prompt]> dir /b build parent.cp310-win_amd64.pyd [prompt]> [prompt]> :: Success! But its name doesn't seem right. Try importing in Python pc032 [prompt]> "e:\Work\Dev\VEnvs\py_pc032_03.10_test0\Scripts\python.exe" -c "import sys;print(sys.version);from parent.child import hello;print(hello());print(\"Done.\n\")" 3.10.9 (tags/v3.10.9:1dd9be6, Dec 6 2022, 19:43:38) [MSC v.1934 32 bit (Intel)] Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'parent' [prompt]> [prompt]> :: Try importing in Python pc064 (this also seems wrong) [prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts\python.exe" -c "import sys;print(sys.version);from parent.child import hello;print(hello());print(\"Done.\n\")" 3.10.9 (tags/v3.10.9:1dd9be6, Dec 6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)] Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: DLL load failed while importing parent: %1 is not a valid Win32 application.
Notes:
The .pyd name is the same as for pc064 case, which is not correct (this is why I used 2 dirs (check note #1.), otherwise the pc064 .pyd would have been overwritten). Check:
Things are definitely wrong, could be a SetupTools bug (maybe fixed in later versions), this might not be the case on your side. Simplest way is to rename the .pyd (there might also be a setup.py argument?).
Output (continued):
[prompt]> ren parent.cp310-win_amd64.pyd parent.cp310-win32.pyd [prompt]> [prompt]> dir /b build parent.cp310-win32.pyd [prompt]> [prompt]> :: Try importing in Python pc032 [prompt]> "e:\Work\Dev\VEnvs\py_pc032_03.10_test0\Scripts\python.exe" -c "import sys;print(sys.version);from parent.child import hello;print(hello());print(\"Done.\n\")" 3.10.9 (tags/v3.10.9:1dd9be6, Dec 6 2022, 19:43:38) [MSC v.1934 32 bit (Intel)] Hi, World! Done.
Worked like a charm!
This is the most generic way of doing things, and it should work for any target build (pc032, aarch64, arm32), once the cross build tools and the appropriate .lib files are present on the host machine.
But, if aiming for pc032 (win32) only, there's a shortcut:
Download and install the pc032 Python (which runs on pc064 machines)
Use it (like in #3.) to build natively
- matplotlib contourf: get Z value under cursor
- Detect if an OCR text image is upside down
- How to sort a list of lists of lists by the total number of items?
- What is the recommended way for retrieving row numbers (index) for polars?
- How to upload an image file to Github using PyGithub?
- How to store and load a Python dictionary with HDF5
- Prevent Selenium from taking the focus to the opened window
- Subclass super().__init__(*args, **kwargs) not working. Says object.__init__() takes exactly one argument (the instance to initialize) when it doesn't
- Overriding python threading.Thread.run()
- When searching for strings in a dataframe via lambda function, how do I resolve the IndexError if no results are found in any column or row?
- ValuerError: Found input variables with inconsistent numbers of samples
- sqlite3 or CSV files
- isPrime Function for Python Language
- Sending data and files as multipart request using aiohttp
- Official repository of Unicode character names
- How to create child window and communicate with parent in TkInter
- with pytest.raises(Exception) not working with flask app.post
- ModuleNotFoundError with pyinstaller but fine in Pycharm
- Storing and retreiving data with Milvus and Langchain
- Python Barplot to represent Ranges
- Docker image python-alpine fails when using lib
- create azure functionapp from docker image via cli
- Routing submodule functions using FastAPI's APIRouter
- clone element with beautifulsoup
- Sharing variables between two seperatly running python scripts
- How do I filter a table by clicking on a bar chart segment?
- pyarrow: find diff for chunkedarray
- Skip statements in case of last item in iterator
- csv file to host.yaml for Nornir
- When I'm plotting a colorbar in python using matplotlib I'm getting an error