Python C extension packaging DLL along with pyd

Question

My project exposes a static library (call it static.lib) to CPython (3.8) interpreter. It consists of a static library that is in turn dependent on a DLL FTDI driver. After reading this thread it appears that optimal solution to providing third party DLLs is to bundle them along with a Python package - to make sure that DLL is located in the same directory as .pyd binary.

The issues I am having is that after running pip install . for my package, the required DLL (call it required.dll) is placed in site-packages/package/required.dll and the actual C extension library (call it package.pyd) is placed in site-packages/package.pyd.

Since it is not in the same directory when I attempt to use the library in Python I get

ImportError: DLL load failed while importing package: The specified module could not be found.

Below is my setup.py

setuptools.setup(
        name="package",
        version="1.0.0",
        packages=setuptools.find_packages(where="src"),
        package_dir={"": "src"},
        py_modules=[splitext(basename(path))[0] for path in glob("src/*.py")],
        use_scm_version=True,
        package_data={
            "package": [
                "_clibs/libs/required.dll",
            ],
        },
        ext_modules=[
            setuptools.Extension(
                "package",
                include_dirs=["src/package/_clibs/inc"],
                sources=[
                    "src/package/_clibs/src/api.cpp",
                    "src/package/_clibs/src/utils.cpp",
                ],
                library_dirs=[
                    "src/package/_clibs/libs",
                ],
                libraries=["static", "User32"],
                language="c++"
            ),
        ],
    )

Directory layout for the project is as follows:

/
setup.py
.tox
src/
...package/
......wrapper.py
......__init__.py
......_clibs/
.........inc/
.........src/
............api.cpp
............utils.cpp
.........libs/
............required.dll
............static.lib

I also use tox for virtual environment management.

The suggested answers here and here outline a very similar setup.py and the same method of including the DLL - through package_data option. The answers seem to suggest that DLL and .pyd are then placed on the same level which does not happen for me. I can't quite place what it is I am missing to get the same behaviour.

python 3.8.6
setuptools 51.0.0
pip 20.3.1

TL;DR DLL is being placed in a different directory to .pyd binary thus making it invisible to Windows loader

Answer 1

After some digging, I found a way that worked for me. This thread has shed light on the problems of DLL loading on Windows and most recent (Python 3.8) developments on the issue.

The solution I went with was borrowed from numpy. In order to properly bundle DLLs along with your C extension:

1. Create a directory within your package that will contain all required DLLs your extension will be using
2. Modify your build procedure to include this directory along with sdist and wheel distributions
3. Once user imports your package, first thing you do is dynamically modify paths for where DLLs will be searched for (two different methods depending if you are on 3.8 or lower)

Roughly, adding package_data to your setup.py should do the trick (minus the shenanigans that come with MANIFEST files and using package_data, read more here)

 package_data={"your package name": ["path_to_DLLs/*"]},

To implement #3, as an option in your __init__.py for the package add the following (taken 99% line by line from numpy __config__.py that gets auto-generated by their very complex build system.

import os
import sys

PATH_TO_DLL = "YOUR DLL DIRECTORY IN YOUR PACKAGE"
extra_dll_dir = os.path.join(os.path.dirname(__file__), PATH_TO_DLL)

if sys.version_info >= (3, 8):
    os.add_dll_directory(extra_dll_dir)
else:
    # legacy DLL loading mechanism through PATH env variable manipulations
    os.environ.setdefault("PATH", "")
    os.environ["PATH"] += os.pathsep + extra_dll_dir

Any feedback is greatly appreciated. The thread linked to this ticket talks about the need for better documentation for C extension onboarding and I have not been able to find any. So far, C extensions + Windows + setuptools made for incredibly frustrating experience.