Build VTK with Python Bindings and CUDA, TBB, & MPI on Windows

Problem

I cannot load VTK into python on Windows with support for CUDA, MPI, and TBB. Volume rendering is unreasonably slow for production with the default PyPI distribution (hours or more) and I need to leverage my hardware to speed up computations.

I have successfully built VTK 9.3.0 for Python 3.8.10-x64 with CMake and Visual Studio and have bin, lib, include, and share folders and have added these to PATH, but Python does not see vtk:

Python 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import vtk
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'vtk'
>>>

I attempted to place the generated Lib/site-packages, bin, lib, and include folders in appropriate places in C:\Program Files\Python38, but I still get a DLL load error:

> py -3.8
Python 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import vtk
Traceback (most recent call last):
  File "C:\Program Files\Python38\lib\site-packages\vtkmodules\__init__.py", line 13, in <module>
    from . import vtkCommonCore
ImportError: DLL load failed while importing vtkCommonCore: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\Python38\lib\site-packages\vtk.py", line 30, in <module>
    all_m = importlib.import_module('vtkmodules.all')
  File "C:\Program Files\Python38\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "C:\Program Files\Python38\lib\site-packages\vtkmodules\__init__.py", line 15, in <module>
    import _vtkmodules_static
ModuleNotFoundError: No module named '_vtkmodules_static'
>>>

Setup

OS: Windows 10 Enterprise, x64-bit, Build 1909
CPU: 2x Intel(R) Xeon(R) Gold 6248R
Disk: 2TB NVMe M.2 SSD
RAM: 192 GB DDR4
Compute GPUs: 2x NVIDIA Quadro RTX8000 in TCC mode with NVLink
Display GPU: 1x NVIDIA Quadro RTX4000
Visual Studio: Visual Studio 16 2019 Community
Windows SDK: 10.0.19041.0, targeting 10.0.18363
Python: 3.8.10 x64-bit
CMake: 3.21.1 (I use CMake-GUI)
vcpkg: 2021-08-12
VTK Source: vtk-9.0.3-cp38-cp38-win_amd64 (download from VTK website; not GitHub master)
VTKPythonPackage: GitHub Link (...not sure if I need this...)
CUDA Toolkit: 11.4
TBB: 2021.3.0 (via Intel(R) oneAPI Base Toolkit)
MPI: 2021.3.0 (via Intel(R) oneAPI HPC Toolkit)

NOTE: If desired, Embree, OSPRay, and OpenVKL are available in the Intel(R) oneAPI Rendering Toolkit

Preparation

NOTE: There are A LOT of programs to be downloaded, installed, and configured, and they are important for proper building. In an attempt to shorten this post to an MRE (Minimal Reproducable Example), please go straight to the Steps to Reproduce section.

NOTE: When setting Environment Variables, you may get an error that a PATH length greater than 2047 is not permitted. To overcome this hurdle, it is simplest to edit the path directly in the Windows Registry. Open regedit as an Admin, navigate to

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment

and set the Path variable there.

TIP: To edit PATH, I copy and paste from the Registry into Word, then selected and replace ;'s with paragraph endings ^p by selecting Ctrl+H and then using the Replace tool as follows:

When finished, I simply reverse the changes by replacing ^p's with ;s. This is simpler for me than struggling with regular expressions.

1. Install Visual Studio

Download the Visual Studio Installer
Run the installer, choose Visual Studio Community 2019, and select Desktop development with C++. Ensure the following are selected in Individual Components

Windows 10 SDK
C++ ATL for latest...
C++ MFC for latest...
C++ CMake tools for Windows

NOTE: (For me on Windows 10 with Visual Studio 2019, these are v142)

Ensure the path to the corresponding bin, lib, and include folders are in the PATH of System Environment Variables. These can be either:

C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\lib
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include

and/or

C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64\
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\lib\x64\
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include

NOTE: (I recommend the latter at a minimum. )

Ensure the following additional System Environment Variables have been set by the installer:

VS140COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\
VS160COMCOMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\Tools\
VS2019INSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community
VSSDK140INSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VSSDK\

NOTE: Even though the installer downloads these to Program Files (x86), they are in fact the x64-bit programs (I know...Windows is weird...)

2. Install CUDA

Download and run cuda_11.4.1_471.41_win10.exe (link)

TIP: The local installer will take longer to download, but your installs will run faster. I recommend saving the installer for the future anyway in case your CUDA Toolkit becomes corrupted or you need to install on another machine.

Ensure the following are added to PATH in System Environment Variables

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\libnvvp

ASIDE: You MUST use version 11.4 for VTK 9. However, as an aside, when using 10.2 in CMake, if CMake cannot find CUDA, this is probably because the CUDA Toolkit 10.2 installer for some reason does not copy the Visual Studio MSBuild Extensions over to your Visual Studio folder. Copy the files at

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\extras\visual_studio_integration\MSBuildExtensions\

C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\VC\VCTargets\BuildCustomizations

NOTE: If you want to run your GPU in TCC mode (WARNING: This disables graphics output), and/or if you have two GPUs connected via NVLink and want to activate NVLink, run the following command in command prompt:

> nvidia-smi -i 0 -dm 1
> nvidia-smi -i 1 -dm 1

Assuming you have two GPUs with ID's 0 and 1 you want to run in TCC mode. (To check, simply run nvidia-smi and see which GPUs you want to activate in TCC mode with the -dm switch). The PugetSystems Instructions have change with CUDA 11. NVIDIA's newer drivers, which ship with the NVIDIA Control Panel, no longer have the Configure SLI, Surround, PhysX under 3D Settings, particularly if using a Quadro card rather than a GeForce card. Instead, NVLink will be activated once TCC mode is set, which disables graphics output. The program they have provided only works for CUDA 10, so you will get an error with it when using CUDA 11 even though it is enabled.

NOTE: If you get an error "Intel Graphics Driver for Windows not found", ignore it. You only need the drivers for the CPU.

NOTE: Don't add more than the /lib/ and /libnvvp/ folders to path. CMake might not be able to find CUDA if you do. If you have to add other folders (e.g. CUPTI), I recommend the lib/ and libnvvp/ folder appear FIRST in PATH.

3. Install `CMake`

Download and install CMake (...that was easy)

4. Install `vcpkg`

Download and configure vcpkg:

cd C:\
git clone https://github.com/microsoft/vcpkg
cd vcpkg
.\bootstrap-vcpkg.bat
vcpkg integrate install

Set the default triplet in the System Environment Variables to x64-windows:

VCPKG_DEFAULT_TRIPLET=x64-windows

Install ports using vcpkg install. Press Configure in CMake and every time it cannot find a dependency you want installed, search for the dependency with vcpkg search and install it with vcpkg install (to remove a package, use vcpkg remove)._

NOTE: BE AWARE that depending on what you install, these dependencies take A LOT of space and take A LONG TIME to download.

5. Install `Intel oneAPI Toolkits`

NOTE: These include TBB, MPI, OSPRay, and the Intel Fortran compiler ifort.exe, which you need.

Intel(R) oneAPI Base Toolkit
Intel(R) oneAPI HPC Toolkit
Intel(R) oneAPI Rendering Toolkit (for Embree, OSPRay, and OpenVKL, optionally)

and add the corresponding lib, bin, and include folders to PATH.

6. Install `Intel SPMD Compiler`

Intel(R) Implicit SPMD Program Compiler

7. Additional Dependencies with Installers

NOTE: You may not need or want need these. I am adding them here for completeness. If CMake complains about not finding these, I recommend installing them.

8. Dependencies that are Unsuccessful on Windows

NOTE: I have tried installing binaries for these and adding them to PATH, but they just don't seem to work on Windows..

NOTE: For each of these, you can also try downloading corresponding python 3.8 zip source file (on GitHub, under Releases, and on conda forge, under Files) to a folder on C:` and then add the subfolders bin, lib, include (and share if available) to the PATH System Environment Variable.

Steps to Reproduce

I set my PATH first. VTK will make bin, include, lib, and share folders, so I make sure these are at the front of PATH. Here is what I have (HINT: Refer to setting PATH in regedit to avoid the Windows error of PATH length greater than 2047 characters in the Preparation section above)
Open an x64 Native Tools Command Prompt for VS 2019 terminal as an Administrator and run

> cmake-gui

This ensures vcvarsall.bat is run and the corresponding Environment Variables it sets are passed on to CMake-GUI.

Set the Source folder to C:\VTK\src and Build folder to C:\VTK

UPDATE 27-AUG-2021:

Step 3 was the problem. The build folder was previously C:\VTK\build. It needed to be C:\VTK. I corrected this.

Press Configure and select Visual Studio 16 2019 as the generator with x64 as the optional platform:

NOTE: I do not know if libraries for VTK and CUDA should be built for 32-bit. I am using 64-bit as my best guess.

After Configuration completes, you will see many variables, which is understandably overwhelming and it is unclear what the differences are between DEFAULT, YES, NO, WANT, and DON'T WANT. Unfortunately, the answer is hidden tucked away in VTK's CMake Module API Doxygen docs

VTK CMake Module API

For now, I leave everything default and edit the following CMake variables:

TIP: Check the Grouped and Advanced check boxes next to the Add Entry button to see all variables grouped by corresponding project/module:

CMAKE_CXX_MP_FLAG    Checked
CMAKE_INSTALL_PREFIX    C:/VTK
VTK_GROUP_ENABLE_Imaging    WANT
VTK_GROUP_ENABLE_MPI    WANT
VTK_GROUP_ENABLE_Qt    WANT
VTK_PYTHON_VERSION    3
VTK_SMP_IMPLEMENTATION_TYPE    TBB
VTK_USE_CUDA    Checked
VTK_USE_MPI    Checked
VTK_WRAP_PYTHON    Checked
VTK_ENABLE_KITS    Checked

and delete

Python2_EXECUTABLE

under Ungrouped Entries

Rerun Configure
Change the following CMake Variables:

Python3_EXECUTABLE    C:/Program Files/Python38/python.exe
Python3_INCLUDE_DIR    C:/Program Files/Python38/include
Python3_LIBRARY    C:/Program Files/Python38/libs/python38.lib

Rerun Configure

Output from 3rd Configure:

Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.18363.
Found Python3: C:/Program Files/Python38/python.exe (found suitable version "3.8.10", minimum required is "3.2") found components: Interpreter Development.Module Development.Embed 
Configuring done

My CMake variables now look like this:

ASIDE: Would be nice if the CMake GUI had an option to output variables to a text file, especially for debugging...

ASIDE: Would be nice if CMake GUI could tell you what had actually been activiated/used since everything shows up with DEFAULT value and we don't want to change that...

ASIDE: Would be nice if there was a high level overview somewhere of what all of these modules entail so users can understand what they want/need...

Click Generate

Output from Generate

Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.18363.
Configuring done
Generating done

Open Visual Studio 2019 as an Administrator
Open the Solution file VTK.sln in C:\VTK\build
Change the build type from Debug to Release for speed:

Right-click on ALL_BUILD and select Build. In the end, the output is

========== Build: 398 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========

Right-click on INSTALL and select Build. In the end, the output is:

========== Build: 116 succeeded, 0 failed, 283 up-to-date, 0 skipped ==========

Issue

Please see problem at top of post.

Update: 26-AUG-2021

Per @JensMunk suggestion, I copied the VTK master branch source code to overwrite vtkmodules/__init__.py with support for os.add_dll_directory(), but to no avail:

r"""
Currently, this package is experimental and may change in the future.
"""
from __future__ import absolute_import

import sys


def _windows_dll_path():
    import os

    _vtk_python_path = 'Lib/site-packages/vtkmodules'
    _vtk_dll_path = 'bin'
    # Compute the DLL path based on the location of the file and traversing up
    # the installation prefix to append the DLL path.
    _vtk_dll_directory = os.path.dirname(os.path.abspath(__file__))
    # Loop while we have components to remove.
    print("Folder Search:")
    print("==============")
    while _vtk_python_path not in ("", ".", "/"):
        # Strip a directory away.
        _vtk_python_path = os.path.dirname(_vtk_python_path)
        _vtk_dll_directory = os.path.dirname(_vtk_dll_directory)
        if _vtk_python_path != "C:\\":
            print(_vtk_python_path)
    print("==============")
    print()
    _vtk_dll_directory = os.path.join(_vtk_dll_directory, _vtk_dll_path)
    print(f"VTK DLL Directory: {_vtk_dll_directory}")
    print()
    print("Adding DLLs...")
    if os.path.exists(_vtk_dll_directory):
        # We never remove this path; it is required for VTK to work and there's
        # no scope where we can easily remove the directory again.
        print(f"DLL: {_vtk_dll_directory}")
        _ = os.add_dll_directory(_vtk_dll_directory)

    print()
    print("Build tree...")
    # Build tree support.
    try:
        from . import _build_paths

        # Add any paths needed for the build tree.
        for path in _build_paths.paths:
            if os.path.exists(path):
                print(f"Path: {path}")
                _ = os.add_dll_directory(path)
    except ImportError:
        # Relocatable install tree (or non-Windows).
        print("Passing: Relocatable install tree")
        pass
    print()
    print("Imported!")


# CPython 3.8 added behaviors which modified the DLL search path on Windows to
# only search "blessed" paths. When importing SMTK, ensure that SMTK's DLLs are
# in this set of "blessed" paths.
if sys.version_info >= (3, 8) and sys.platform == "win32":
    _windows_dll_path()


# ------------------------------------------------------------------------------
# this little trick is for static builds of VTK. In such builds, if
# the user imports this Python package in a non-statically linked Python
# interpreter i.e. not of the of the VTK-python executables, then we import the
# static components importer module.
try:
    from . import vtkCommonCore
except ImportError:
    import _vtkmodules_static

# ------------------------------------------------------------------------------
# get the version
__version__ = "9.0.3"

Attempting to import again, I get the same error, so DLLs are still missing:

Python 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import vtk
Folder Search:
==============
Lib/site-packages
Lib

==============

VTK DLL Directory: C:\Program Files\Python38\bin

Adding DLLs...
DLL: C:\Program Files\Python38\bin

Build tree...
Passing: Relocatable install tree

Imported!
Traceback (most recent call last):
  File "C:\Program Files\Python38\lib\site-packages\vtkmodules\__init__.py", line 69, in <module>
    from . import vtkCommonCore
ImportError: DLL load failed while importing vtkCommonCore: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\Python38\lib\site-packages\vtk.py", line 30, in <module>
    all_m = importlib.import_module('vtkmodules.all')
  File "C:\Program Files\Python38\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "C:\Program Files\Python38\lib\site-packages\vtkmodules\__init__.py", line 71, in <module>
    import _vtkmodules_static
ModuleNotFoundError: No module named '_vtkmodules_static'

Solution

I was putting my generated CMake files into C:\VTK\build\ and then compiling and installing in C:\VTK, which is why Python couldn't find the DLLs. I have corrected the corresponding step in my post so the install instructions are complete and correct.

See section Steps to Reproduce, step 3