Search code examples
pythonpipgitlabpypigitlab-api

How to setup two PyPI indices

I have a local GitLab installation that comes with a local PyPI server to store company internal Python packages.

How can I configure my PyPI to search packages in both index servers?

I read about .pypirc / pip/pip.ini and found various settings but no solution so far.

  1. Most solutions permanently switch all searches to the other index server. But I want to be able to install and update packages from pypi.org as normal while some packages come from the local index.
  2. setting multiple index servers with credentials seams to be limited to distutils (used e.g. by twine) only, but is not used by pip
  3. There is confusion if to configure index servers in [global] or [install]. I assume the latter one is a rule subset for pip install. (The documentation is here unclear.)
  4. While twine can reference a repository entry in the config file like -r gitlab refers to a [gitlab] section, such a named reference can't be used by pip...

So what I want to achieve:

  • pip should be able to install and update regular packages from pypi.org like colorama
  • pip should be able to install and update packages from gitlab.company.com
    • authentication with username (__token__) and password (7a3b62342c784d87) must work

Experiment so far:

[global]


[install]
find-links =
    https://pypi.org
    https://gitlab.company.de/api/v4/projects/2142423/packages/pypi
trusted-host =
    https://pypi.org
    https://gitlab.company.de/api/v4/projects/2142423/packages/pypi

[distutils]
index-servers =
    gitlab

[gitlab]
repository = https://gitlab.company.de/api/v4/projects/2142423/packages/pypi
username = __token__
password = geheim
Solution

  • Goal

    1. pip install should install/update packages from GitLab as well as PyPi repo. If same package is present in both, PyPi is preferred.
    2. pip install should support authentication. Preferred, if somehow we can make it read from a config file so that we don't need to specify it repeatatively.

    Theory

    1. pip install supports --extra-index-url to specify additional PyPi indexes. The same can also be provided via pip.conf file.
    2. pip uses requests which supports ~/.netrc as config file (docs).

    Steps

    1. Create a pip.conf (pip.ini if on Windows) in any of the locations suggested by pip config -v list.
    2. Add your GitLab PyPi index URL to pip.conf.
    [install]
extra-index-url = https://gitlab.com/api/v4/projects/12345678/packages/pypi/simple
    1. Create or update your ~/.netrc file and add your auth details for GitLab.
    machine gitlab.com
    login <token-name>
    password <token-pass>
    1. We can now install packages as simply as pip install <package-name>. pip will now look at both indexes to find your packages, with preference provided to the one pointed by index-url.

    Additional info

    1. The same could have been possible for pip search too, had there been support for multiple indexes. Till then, one needs to manually specify which PyPi index URL should be used. GitLab does not seem to support pip search since it throws 415 Client Error: Unsupported Media Type when specified as the PyPi index.
    2. As for your doubts, each section in pip.conf points to that particular command, [install] provides configuration for pip install, [search] for pip search and so on. [global] probably refers to parameters that can be specified for all the commands be it pip install or pip search.
    3. .pypirc file is made specially for configuring package indexes related to upload (used by twine/flint), where as pip.conf is associated with configuring pip which manages python packages on your local system.