Why does Pip disregard configured repository with nested dependencies?

Problem

Let us say I have a completely empty Python+Pip+R (pip 19.3.1) environment on a Linux machine and I want to install the package rpy2 with pip. Since I am behind a corporate firewall I configure pip to use a private repository.

[global]
index-url = http://private.com/artifactory/api/pypi/PyPI/simple
trusted-host = private.com

Now I execute pip install rpy2 and I will get back the following error:

Couldn't find index page for 'cffi'
 Download error on https://pypi.python.org/simple/

So pip tries to resolve the nested dependency by looking and installing cffi from the official PyPi repository. It completely ignores the repo I have configured.

When I run pip install cffi && pip install rpy2 one after another everything works as expected.

Here is the full error output:

ERROR: Command errored out with exit status 1:
 command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-8vuadu93/rpy2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-8vuadu93/rpy2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-8vuadu93/rpy2/pip-egg-info
     cwd: /tmp/pip-install-8vuadu93/rpy2/
Complete output (25 lines):
Download error on https://pypi.python.org/simple/cffi/: [Errno -2] Name or service not known -- Some packages may not be found!
Couldn't find index page for 'cffi' (maybe misspelled?)
Download error on https://pypi.python.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
No local packages or working download links found for cffi>=1.13.1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-8vuadu93/rpy2/setup.py", line 183, in <module>
    'rinterface_lib/R_API_eventloop.h']}
  File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 128, in setup
    _install_setup_requires(attrs)
  File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 123, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 513, in fetch_build_eggs
    replace_conflicting=True,
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 774, in resolve
    replace_conflicting=replace_conflicting
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1057, in best_match
    return self.obtain(req, installer)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1069, in obtain
    return installer(requirement)
  File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 580, in fetch_build_egg
    return cmd.easy_install(req)
  File "/usr/lib/python3/dist-packages/setuptools/command/easy_install.py", line 692, in easy_install
    raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('cffi>=1.13.1')
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Question

Is this a bug in Pip, a specific problem with rpy2 or am I missing something?

Updates

Running the following commands gives me the same error:

pip install rpy2 --no-index --find-links http://private.com/artifactory/api/pypi/PyPI/simple
pip install rpy2 --index-url http://private.com/artifactory/api/pypi/PyPI/simple

I used -vvv and it seems like the problem occurs somewhere inside of setuptools

Solution

The package rpy2 uses setuptools which again uses easy_install.py. It also works with an index_url variable. But it gets the value not from pip.config but distutils.cfg.

I identified all Python versions I have installed with find / -name "distutils". Then I added the a distutils.cfg with the following content to each of these directories:

[easy_install]
index_url = blablabla

And now it works, I execute pip install rpy2 and all missing requirements are installed in one go

Solution

I believe it could be caused by the fact that cffi is listed as setup_requires in rpy2's setup.py. Most likely because cffi is required to build the project itself before it can be installed. This kind of build dependencies are not handled by pip directly, so its index-url option has no effect.

The solution is to tell setuptools about the alternative index in a distutils configuration file

[easy_install]
index_url = https://my.index-mirror.com

References: