Avoiding package reinstalls when developing binary module

Question

I'm working on a Python module that contains compiled code. Compiling and installing the module works when using pip install . from the directory containing pyproject.toml and setup.py (I'm not actually sure which combination of these files I need, but it works when both are present at least).

The issue I'm having is that every time I run pip install ., pip will reinstall cmake and friends:

From the output: Successfully installed cmake-3.28.3 distro-1.9.0 ninja-1.11.1.1 packaging-23.2 scikit-build-0.17.6 setuptools-69.1.1 wheel-0.42.0

This has the effect that every time I'm iterating on changes in the source code, I have to sit through this reinstall which takes several seconds each time.

Is there not a way to install just my module and none of the dependencies? I've tried pip install --no-deps . and pip install --no-dependencies . to no avail.

Contents of pyproject.toml:

[build-system]
requires = [
    'setuptools>=42',
    'scikit-build>=0.13',
    'cmake>=3.20',
    'ninja'
]
build-backend = 'setuptools.build_meta'

setup.py just has the name of the package, license, etc.

Answer 1

When pip sees an pyproject.toml file, it automatically enables the "build isolation" feature[^1].

This means that pip will create a "disposable virtual environment" just for building your project and delete this environment once the build is done[^2].

That is why the build dependencies are always re-installed.

To avoid this, you can perform the installation using following steps:

1. Manually install the build dependencies in the same environment where you want your package to be installed (it is a good idea to use virtual environments here).
   • In your example, this means running:

     pip install ninja cmake>=3.20 scikit-build>=0.13
     pip install -U setuptools wheel  # It is good to use the latest version...
     

2. Explicitly disable build isolation when installing your project:

   pip install --no-build-isolation .
   

   • Note that when using --no-build-isolation pip will not install any build dependency automatically. That is why we need step 1.
   • This is also why you need to install wheel (setuptools automatically adds wheel to whatever you specify in build-system.requires).

[^1]: Eventually, future versions of pip will enable "build isolation" by default, even if you don't have a pyproject.toml.

[^2]: The reasoning behind this behaviour is, broadly speaking, to avoid dependency hell and version collisions.