Search code examples
pythonc++python-3.xboost-python

Python failing to load boost.python dll

I'm having some problems with a trivial boost python setup. I have seen a lot of other people have had problems, but none of them seem to be the same issue as mine, as none of their resolutions worked. For reference, I am on windows 10, using mingw64 10.2 as part of msys2 for my c++ compiler. I built boost using that compiler for debug and optimised, and I built a dll linking against boost.python with that compiler as well.

my Cmake file:

cmake_minimum_required(VERSION 3.20)
 
project(test)
set(CMAKE_CXX_STANDARD 20)

find_package(Python3 REQUIRED COMPONENTS Interpreter Development) 
include_directories(${Python3_INCLUDE_DIRS})


set(Boost_ARCHITECTURE -x64)
set(Boost_NO_WARN_NEW_VERSIONS ON) 
# set(Boost_DEBUG ON)
set(Boost_INCLUDE_DIR "C:/Devel/install/include/boost-1_76")
set(Boost_LIBRARY_DIR "C:/Devel/install/lib")
SET(Boost_USE_STATIC_LIBS OFF)
SET(Boost_USE_STATIC_RUNTIME OFF)
add_definitions("-DBOOST_ALL_DYN_LINK")
add_definitions("-DBOOST_UUID_USE_SSE2")
add_definitions("-DBOOST_UUID_USE_SSE3")
add_definitions("-DBOOST_PYTHON_STATIC_LIB")
SET(CMAKE_INCLUDE_PATH ${CMAKE_INCLUDE_PATH} "C:/Devel/install/include")
SET(CMAKE_LIBRARY_PATH ${CMAKE_LIBRARY_PATH} "C:/Devel/install/lib")

find_package(Boost 1.76.0 REQUIRED COMPONENTS python39)
include_directories(${Boost_INCLUDE_DIRS})

file(GLOB_RECURSE PythonBindings_SOURCES CONFIGURE_DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/src/PythonBindings/*.cpp")
add_library(PythonBindings SHARED ${PythonBindings_SOURCES})
target_link_libraries(PythonBindings Boost::python39 Python3::Module)
set_target_properties(PythonBindings PROPERTIES LINKER_LANGUAGE CXX PREFIX "" SUFFIX ".pyd" IMPORT_PREFIX "" IMPORT_SUFFIX ".pyd.a")
target_compile_definitions(PythonBindings PRIVATE EXPORT_PYTHONBINDINGS)

Cpp file:

#include <boost/python.hpp>

char const* 
helloWorld()
{
  return "Hello, world!";
}

using namespace boost::python;

BOOST_PYTHON_MODULE(PythonBindings)
{
  def("hello_world", helloWorld);
}

This successfully compiles to 'PythonBindings.pyd'. Opening this in dependency walker, I can see it exports the symbol 'PyInit_PythonBindings'

When I try use this dll from python (python -vv py/helloworld.py)

import PythonBindings;

PythonBindings.hello_world()

I get:

Traceback (most recent call last):
  File "C:\Devel\Working\test\py\helloworld.py", line 18, in <module>
    import PythonBindings;
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 565, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1173, in create_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
ImportError: DLL load failed while importing PythonBindings: The specified module could not be found.

I am not sure what the problem is. My $PATH variable contains the search paths for all the dependencies of PythonBindings.pyd (libboost_python39-mgw10-mt-x64-1_76.dll, kernel32.dll, MSVCRT.dll, libgcc_s_seh-1.dll, libstdc++-6.dll, python39.dll)

I named the dll the same as the python module, which I know some people have had trip them up, and I used the same c++ compiler and python version for building boost and my library, and I used the same python version for linking both of those as I did for running my python script.

I'm all out of ideas on how to fix this issue.

Solution

  • I followed the tutorial here following doqtor's comment, and found that the problem was python not loading several dlls that were runtime dependencies of PythonBindings.pyd.

    This was fixed by adding

    import sys
import os
[os.add_dll_directory(dir) for dir in sys.path if os.path.isdir(dir)]

    before import PythonBindings

    It's very odd to me that python doesn't search sys.path for dll dependencies. The paths that it needed to search were also on os.environ['PATH'], but again it didn't search them. I'm not sure if there is a nicer way to do this. I would like to hear about it if there is.