When I try to compile a Cython project with submodules using the gmp library and including C ++ files, I get an error:
ImportError: DLL load failed while importing ...
In particular, I want to wrap in Cython a class written in C++, which use GMP, by using Mingw64 as compiler. I use Python3.8 and Mingw64 has been is installed by means of MSYS2.
In the following, I provide you with a minimal reproducible example (as asked by @ead).
The directory tree is as follows:
project
setup.py
pytest.py
include
test.h
test.cpp
test
submodule
cy_test.pxd
cy_test.pyx
where, in project/include, I putted the class written in C++ I want to wrap.
project/include/test.h is
#ifndef TEST_LIB
#define TEST_LIB
#include <gmpxx.h>
using namespace std;
class Test
{
private:
mpz_t n;
public:
Test();
Test(const char* expr);
virtual ~Test()
{
mpz_clear(this->n);
}
};
#endif // TEST_LIB
whereas project/include/test.cpp is
#ifndef TEST_IMPL
#define TEST_IMPL
#include <iostream>
#include <cstdarg>
#include <test.h>
Test::Test()
{
mpz_init_set_ui(this->n, 0);
}
Test::Test(const char* expr)
{
mpz_init_set_str(this->n, expr, 10);
}
#endif // TEST_IMPL
project/test/submodule/cy_test.pxd is:
cdef extern from "test.cpp":
pass
# Declare the class with cdef
cdef extern from "test.h":
cdef cppclass Test:
Test() except +
Test(char* n) except +
Instead, project/test/submodule/cy_test.pyx is:
from test.submodule.cy_test cimport Test
cdef class PyTest:
cdef Test* n
def __cinit__(self, object n):
cdef char* string = n
self.n = new Test(string)
def __dealloc__(self):
del self.n
The file project/pytest.py is simply an import of cy_test:
from test.submodule import cy_test
Finally, project/setup.py is:
import os
import sys
from setuptools import find_packages
from distutils.core import setup
from distutils.command.clean import clean
from distutils.sysconfig import get_python_inc
from distutils.command.build_ext import build_ext
from Cython.Build import cythonize
from Cython.Distutils import Extension
os.environ["CC"] = "g++"
os.environ["CXX"] = "g++"
MINGW_DIR = "C:/msys64/mingw64"
CWD = os.getcwd()
BASE_DIRS = [os.path.join(CWD, "include"), os.path.join(CWD, "test"), get_python_inc(), "C:/Program Files/Python38"]
INCLUDE_DIRS = [os.path.join(MINGW_DIR, "include")] + BASE_DIRS
LIB_DIRS = [os.path.join(MINGW_DIR, "lib")]
EXTRA_ARGS = ["-O3", "-std=c++17"]
EXTRA_LINK_ARGS = []
LIBRARIES = ["gmp", "gmpxx", "mpc", "mpfr"]
EXTRA_LIBRARIES = []
ext = [
Extension(
name="test.submodule.cy_test",
sources=["./test/submodule/cy_test.pyx"],
language="c++",
include_dirs=INCLUDE_DIRS,
library_dirs=LIB_DIRS,
libraries=LIBRARIES,
extra_link_args=EXTRA_LINK_ARGS,
extra_compile_args=EXTRA_ARGS,
extra_objects=EXTRA_LIBRARIES,
cython_cplus=True,
cython_c_in_temp=True)
]
setup(
name="test",
packages=find_packages(),
package_dir={
"test": "test",
"test/submodule": "test/submodule"},
include_package_data=True,
package_data={
'test': ['*.pyx', '*.pxd', '*.h', '*.c', '*.cpp', '*.dll'],
'test/submodule': ['*.pyx', '*.pxd', '*.h', '*.c', '*.cpp']
},
cmdclass={'clean': clean, 'build_ext': build_ext},
include_dirs=BASE_DIRS,
ext_modules=cythonize(ext,
compiler_directives={
'language_level': "3str",
"c_string_type": "str",
"c_string_encoding": "utf-8"},
force=True,
cache=False,
quiet=False),
)
I compile setup.py as python setup.py build_ext --inplace --compiler=mingw32 in order to use MinGW and g++.exe as, by default on Windows, the compiler is taken from Visual Studio, which gives me other errors like undefined reference to '__gmpz_init'.
The compilation ends without errors, but when I run the pytest.py script I get the following error:
ImportError: DLL load failed while importing cy_test
I have already dumped cy_test.cp38-win_amd64.pyd as in link and I have got this:
Dump of file cy_test.cp38-win_amd64.pyd
File Type: DLL
Image has the following dependencies:
KERNEL32.dll
msvcrt.dll
api-ms-win-crt-environment-l1-1-0.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-private-l1-1-0.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-time-l1-1-0.dll
libgcc_s_seh-1.dll
libstdc++-6.dll
libgmp-10.dll
python38.dll
Summary
1000 .CRT
1000 .bss
1000 .data
1000 .edata
2000 .idata
1000 .pdata
2000 .rdata
1000 .reloc
4000 .text
1000 .tls
1000 .xdata
How can I solve? I have already seen on cython-github-issues that, by including the following two lines in pytest.py
import os
os.add_dll_directory(mingw64_bin_path)
the above error disappear and the submodule is imported correctly, but I would like a solution that avoids including the Mingw path outside the setup (assuming it exists).
I just accidentally found the solution to the above problem. The problem is the package setuptools (which in my case is the version 60.9.1)! Indeed, by executing python setup.py build_ext --inplace --compiler=mingw32, the latter will call the class Mingw32CCompiler into setuptools/_distutils/cygwinccompiler.py which contains these two lines:
...
shared_option = "-shared"
...
self.dll_libraries = get_msvcr()
These lines will produce the compiling command g++ -shared ... -lucrt -lvcruntime140, that is, the pyd file, generated by this latter command, would be a shared library which needs a lot of dll dependencies in order to be executed. In order to avoid these dependencies, it is mandatory to comment the line self.dll_libraries = get_msvcr(), which as a consequence removes -lucrt -lvcruntime140 from the compilation command. Furthermore, one has to modify the setup.py by substituting EXTRA_LINK_ARGS = [] with EXTRA_LINK_ARGS = ["-static"] in order to get the following compiling command: g++ -shared ... -static, which builds the pyd file as a static library that have no dll dependencies. Indeed, after the above modifications, by dumping cy_test.cp38-win_amd64.pyd we get:
Dump of file cy_test.cp38-win_amd64.pyd
File Type: DLL
Summary
1000 .CRT
1000 .bss
4000 .data
1000 .edata
2000 .idata
C000 .pdata
16000 .rdata
2000 .reloc
E8000 .text
1000 .tls
10000 .xdata
I wonder why MSVCR it is not optional for the MinGW compiler in setuptools...