Tell pip package to install build dependency for its own install and all install_requires installs

Question

I am installing a package whose dependency needs to import numpy inside its setup.py. It also needs Cython to correctly build this dependency. This dependency is scikit-learn==0.21.2. Here is the setup.py of my own package called mypkgname:

from setuptools import find_packages, setup

import Cython  # to check that Cython is indeed installed
import numpy  # to check that numpy is indeed installed

setup(
    name="mypkgname",
    version="0.1.0",
    packages=find_packages("src", exclude=["tests"]),
    package_dir={"": "src"},
    install_requires=[
        "scikit-learn==0.21.2"
    ],
)

To make sure that numpy and Cython are available inside mypkgname's setup.py when pip installs mypkgname, I set up the pyproject.toml like this:

[build-system]
requires = ["setuptools>=40.8.0", "Cython", "numpy>=1.11.0,<=1.22.4", "wheel"]
build-backend = "setuptools.build_meta"

After running pip install -e ., the import numpy; import Cython in mypkgname's setup.py work, but the import Cython inside the scikit-learn==0.21.2 install does not:

    File "/home/vvvvv/.pyenv/versions/3.8.12/envs/withingswpm04-38/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1016, in get_subpackage
      config = self._get_configuration_from_setup_py(
    File "/home/vvvvv/.pyenv/versions/3.8.12/envs/withingswpm04-38/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 958, in _get_configuration_from_setup_py
      config = setup_module.configuration(*args)
    File "sklearn/utils/setup.py", line 8, in configuration
      from Cython import Tempita
  ModuleNotFoundError: No module named 'Cython'
  error: subprocess-exited-with-error

I don't understand why Cython is available for the install of my own mypkgname, but not for the install_requires packages of mypkgname. As if Cython was uninstalled(?) before installing the install_requires packages. But by checking the logs with pip install -v, it does not seem it is the case.

---

I tried installing setuptools and Cython beforehand:

pip install setuptools Cython
pip install -e .

and it works. Indeed, it seems that scikit-learn==0.21.2 needs these 2 packages already installed to be properly installed. However, the scikit-learn version I am trying to install does not specify any Cython build requirements inside a pyproject.toml. Here is a link to the setup.py of the scikit-learn package.

If I just install setuptools, it still fails with the same error as in the first example (ModuleNotFoundError: No module named 'Cython').

Notes:

• I am forced for some out-of-context reasons to use this specific version of scikit-learn.
• I am using Python 3.8.12, pip 23.0.1 and Ubuntu 22.04. I tried this using pyenv and virtualenv with the same results. I also tried with pip install -v -U --no-cache-dir.

---

How can I tell mypkgname that I need numpy, Cython and setuptools installed from the beginning of the pip install until the very end? I want the numpy, Cython and setuptools packages to be available in the install of mypkgname and in every install of the install_requires packages.

Since it will be a package deployed on pypi, I don't want people to have anything other than pip and maybe setuptools already installed when running pip install mypkgname.

Answer 1

For building packages pip uses build Isolation. I.e. it installs build dependencies into a separate virtual environment, build a package and remove the isolating venv. So build dependencies (in your case Cython and numpy) are removed along with the isolating venv.

You can disable isolation but better and more correct way is to declare your dependencies two times — as build dependencies and runtime dependencies:

    install_requires=[
        "scikit-learn==0.21.2",
        "Cython",
        "numpy>=1.11.0,<=1.22.4",
    ],

When building your package pip/setuptools include the list of dependencies into the wheel and they will be automatically installed in the target environment along with your package.