Search code examples
pythonpathdependenciesvirtualenvflake8

Global development dependencies in python, aside from virtual environments


Problem: it's very annoying to have to install common dev dependencies in every python virtual environment (including virtual environments created by pipenv, poetry, etc.). For example, flake8, pylint, jedi, black, etc.

It would be nice to just set those up to be globally available from every virtual environment without having to manually install every time.

The last time someone asked how to do this, they got a bunch of comments scolding them for fighting their tools and no actual answer except a VSCode specific thing.

However, I recently came across a comment in the docs for the Elpy package (a Python package for Emacs) that suggests that such a thing really is possible more globally:

It is possible to create a single virtual env for the sole purpose of installing flake8 in there, and then simply link the command script to a directory inside your PATH, meaning you do not need to install the program in every virtual env separately.

That sounds like an amazing idea... and one that could apply to lots of other devtool-type dependencies. Unfortunately, the people who wrote the Elpy docs didn't go into any details of how such a thing would actually happen.

Can someone help translate the quoted paragraph into actual steps to achieve this goal? Like, "command script?" What's that? Is the idea just that a virtualenv lives in a directory somewhere, and that adding that directory to one's shell $PATH will make the libraries within it available to every other virtualenv somehow? Thanks!


Solution

  • sure -- this is precisely how I set up my machine, including bootstrap steps from nothing:

    # in .bashrc
    export PATH=$HOME/bin:$PATH
    
    # some script you run once
    mkdir -p ~/opt
    curl -o virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz
    python3 virtualenv.pyz ~/opt/venv
    rm virtualenv.pyz
    
    ~/opt/venv/bin/pip install flake8 pre-commit tox twine
    mkdir -p ~/bin
    ln -s ~/opt/venv/bin/flake8 ~/bin/
    ln -s ~/opt/venv/bin/pre-commit ~/bin/
    ln -s ~/opt/venv/bin/tox ~/bin/
    ln -s ~/opt/venv/bin/twine ~/bin/
    # etc.
    

    I first download virtualenv.pyz which is a zipapp so I don't need to invoke any global pip / potentially poorly packaged virtualenv

    then I set up a virtualenv in ~/opt/venv

    inside there I install my tools

    and I symlink them into ~/bin which I've put on my PATH via .bashrc

    My actual code for this lives in my configuration management repository: https://github.com/asottile/personal-puppet/blob/753d9491c27b17ae6a2a145fc2aad25d8896e76f/modules/desktop/manifests/venv.pp