Search code examples
pythoncachingpypi

Python package proxy/cache

I'm trying to find a way to cache or store certain python packages for every computer on my network. I've had a look at the following solutions:

  • pypicache but it is no longer being actively developed and the author recommends devpi, see here: https://bitbucket.org/micktwomey/pypicache/
  • proxypypi but the author here also recommends devpi, see: https://bitbucket.org/r1chardj0n3s/proxypypi
  • devpi - for this I followed one of the quickstart guides but this only allowed access from localhost, so I tried following the permanent quickstart guide. Though I'm not sure about parts in here, namely:
    • The use of virtualenv. The reasoning behind this is not explained, why can't it just be installed normally? (Why would you use virtualenv if you're not a python packager?)
    • The deprecated part of the install instructions (using --gendeploy). There is no alternative option, do I just ignore it? It seems important.
    • Why does it need supervisor?
  • So I decided to have a look at pypiserver, which was super easy to setup, and I can tell pip to use it with the -i option, but it won't seem to cache any packages. If I stop the pypiserver then the pip install fails because it cannot connect to it, so that means it must be using it. Except it's not storing any packages. My best guess here is that it's just proxying the request and not caching the download.

Is there a standard way of doing this? My goals are to protect against loss of internet connectivity or the removal of packages so that the clients on the network can still setup a development or production machine if either of these happen, also to speed up setting up a dev or production machine.

I feel like both devpi and pypiserver are good options, I just seem to be missing something to make them work.

EDIT: It appears I was correct about pypiserver in the fact that is only proxying requests to PyPI. It is for uploading your own packages, so you can download them from pypiserver but if your looking for an official package, you get it from PyPI.

Solution

  • A new version of Devpi was released a few days ago. The latest documentation has cleared up all my issues and I have decided to use devpi-server 2.0.1. Devpi not only provides a caching proxy for PyPI packages, but allows you to upload custom packages using the devpi-client. As I do not need this functionality, I am just using the root/pypi index as shown here. Though I followed the instructions on this page minus the last part about the devpi-client.