Search code examples
pythonpython-3.xpip

Per line index url in requirements.txt

Suppose I have the following PyPIs:

  • public PyPi (standard packages)
  • gitlab pypi (because internal team ABC wanted to use this)
  • artifactory PyPi (because contractor team DEF wanted to use this)

Now suppose package titled "ABC" exists on all of them, but are not the same thing (for instance, "apples," which are 3 entirely different packages on all pypis.). How do I do something in my requirements and setup.py to map the package name to the pypi to use?

Something like:

package_def==1.2.3 --index-url=artifactory
apples==1.08 --index-url=gitlab # NOT FROM PUBLIC OR FROM ARTIFACTORY
package_abc==1.2.3 --index-url=artifactory
package_efg==1.0.0 # public pypi

I don't even know how I'd configure the setup.py in this instance either.

I really don't want multiple requirements.txt with different index urls at the top. I also don't want --extra-index-url due to the vulnerabilities it could introduce when using a private pypi.

I tried googling around, messing around with the order of requirements.txt, breaking it up into different files, etc. No luck. Seems that the last --index-url is always used to install all packages.

Any ideas?

Solution

  • The question gets back to the idea that a package dependency specification usually is a state of need that is independent of how that need should be satisfied.

    So the dependency declaration “foo==1.0.0” (the thing declared as part of the package metadata) means “I need the package named foo with version 1.0.0" and that is in principle implementation independent. You can install that package with pip from PyPI, but you could also use a different tool and/or different source to satisfy that requirement (e.g. conda, installation-from-source, etc.).

    This distinction is the reason why there's no good way to do this.

    There are a few work arounds:

    • You can specify the full link to a wheel you want to pip install
    • You can use an alternative tool like Poetry, which does support this a little more cleanly.

    For my particular usecase, I just listed the full link to the wheel I wanted to pip install, since upgrading to poetry is out of scope at the moment.