How to make Python version executables global across multiple pyenv-virtualenv virtual environments

A pyenv Python version (e.g. 3.10.4) has the "normal" expected Python executables associated with it (e.g., pip, 2to3, pydoc)

$ ls "${PYENV_ROOT}/versions/3.10.4/bin"
2to3       idle   idle3.10  pip3     pydoc   pydoc3.10  python-config  python3-config  python3.10-config
2to3-3.10  idle3  pip       pip3.10  pydoc3  python python3        python3.10      python3.10-gdb.py

and a pyenv-virtualenv virtual environment has only the executables that one would a get inside the virtual environment directory structure

$ pyenv virtualenv 3.10.4 venv
$ ls "${PYENV_ROOT}/versions/venv"
bin  include  lib  lib64  pyvenv.cfg
$ ls "${PYENV_ROOT}/versions/venv/bin/"
Activate.ps1  activate  activate.csh  activate.fish  pip  pip3  pip3.10  pydoc  python  python3  python3.10

by default after creation, the venv virtual environment doesn't know about executables of the Python version that it is associated to, like 2to3

$ pyenv activate venv
(venv) $ 2to3 --help
pyenv: 2to3: command not found

The `2to3' command exists in these Python versions:
  3.10.4

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.

so to allow for a virtual environment like venv to have access to these executables you add both it and the Python that created it to pyenv global so that pyenv will "fall back" to the Python version when the executable isn't found

(venv) $ pyenv deactivate
$ pyenv global venv 3.10.4
(venv) $ pyenv global 
venv
3.10.4
(venv) $ 2to3 --help | head -n 3
Usage: 2to3 [options] file|dir ...

Options:

This pattern works for one virtual environment, but how do you maintain access to executables like 2to3 (or pipx as seen below`) when you have multiple virtual environments in play?

(venv) $ pyenv virtualenv 3.10.4 example && pyenv activate example
(example) $ 2to3  
pyenv: 2to3: command not found

The `2to3' command exists in these Python versions:
  3.10.4

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.

Reproducible example

Using the following Dockerfile

FROM debian:bullseye

SHELL ["/bin/bash", "-c"]
USER root

RUN apt-get update -y && \
    apt-get install --no-install-recommends -y \
        make \
        build-essential \
        libssl-dev \
        zlib1g-dev \
        libbz2-dev \
        libreadline-dev \
        libsqlite3-dev \
        wget \
        curl \
        llvm \
        libncurses5-dev \
        xz-utils \
        tk-dev \
        libxml2-dev \
        libxmlsec1-dev \
        libffi-dev \
        liblzma-dev \
        g++ && \
    apt-get install -y \
        git && \
    apt-get -y clean && \
    apt-get -y autoremove && \
    rm -rf /var/lib/apt/lists/*

# Install pyenv and pyenv-virtualenv
ENV PYENV_RELEASE_VERSION=2.3.0
RUN git clone --depth 1 https://github.com/pyenv/pyenv.git \
        --branch "v${PYENV_RELEASE_VERSION}" \
        --single-branch \
        ~/.pyenv && \
    pushd ~/.pyenv && \
    src/configure && \
    make -C src && \
    echo 'export PYENV_ROOT="${HOME}/.pyenv"' >> ~/.bashrc && \
    echo 'export PATH="${PYENV_ROOT}/bin:${PATH}"' >> ~/.bashrc && \
    echo 'eval "$(pyenv init -)"' >> ~/.bashrc && \
    . ~/.bashrc && \
    git clone --depth 1 https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv && \
    echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc

# Install CPython
ENV PYTHON_VERSION=3.10.4
RUN . ~/.bashrc && \
    echo "Install Python ${PYTHON_VERSION}" && \
    PYTHON_MAKE_OPTS="-j8" pyenv install "${PYTHON_VERSION}"

# Make 'base' virtual envirionment, add it and its Python version to global for
# executables like 2to3 or pipx to be findable
# c.f. https://github.com/pyenv/pyenv-virtualenv/issues/16#issuecomment-37640961
# and then install pipx into the 'base' virtual environment and use pipx to install
# pepotron
RUN . ~/.bashrc && \
    pyenv virtualenv "${PYTHON_VERSION}" base && \
    echo "" && echo "Python ${PYTHON_VERSION} has additional executables..." && \
    ls -lh "${PYENV_ROOT}/versions/${PYTHON_VERSION}/bin" && \
    echo "" && echo "...compared to 'base' virtualenv made with Python ${PYTHON_VERSION}" && \
    ls -lh "${PYENV_ROOT}/versions/base/bin" && \
    echo "" && echo "...because 'base' is actually a symlink" && \
    ls -lh "${PYENV_ROOT}/versions/" && \
    pyenv global base "${PYTHON_VERSION}" && \
    python -m pip --quiet install --upgrade pip setuptools wheel && \
    python -m pip --quiet install pipx && \
    python -m pipx ensurepath && \
    eval "$(register-python-argcomplete pipx)" && \
    pipx install pepotron

WORKDIR /home/data

built with

docker build . --file Dockerfile --tag pyenv/multiple-virtualenvs:debug

it can be run with the following to demonstrate the problem

$ docker run --rm -ti pyenv/multiple-virtualenvs:debug
(base) root@26dfa530cd82:/home/data# pyenv global 
base
3.10.4
(base) root@26dfa530cd82:/home/data# 2to3 --help | head -n 3
Usage: 2to3 [options] file|dir ...

Options:
(base) root@26dfa530cd82:/home/data# pipx list
venvs are in /root/.local/pipx/venvs
apps are exposed on your $PATH at /root/.local/bin
   package pepotron 0.6.0, installed using Python 3.10.4
    - bpo
    - pep
(base) root@26dfa530cd82:/home/data# pep 3.11
https://peps.python.org/pep-0664/
(base) root@26dfa530cd82:/home/data# pyenv virtualenv 3.10.4 example                          
(base) root@26dfa530cd82:/home/data# pyenv activate example
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
(example) root@26dfa530cd82:/home/data# 2to3
pyenv: 2to3: command not found

The `2to3' command exists in these Python versions:
  3.10.4

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.
(example) root@26dfa530cd82:/home/data# pipx
pyenv: pipx: command not found

The `pipx' command exists in these Python versions:
  3.10.4/envs/base
  base

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.

So how can one have something like pipx, that is designed to be installed globally, work globally if it is installed in a pyenv-virtualenv virtual environment so that you don't have to have anything installed with pip in the system Python?

It would seem that instead of ever using pyenv activate to activate a virtual environment you would need to deactivate any virtual environments and then only use pyenv global <virtual environment name> <virtual environment Python version> to effectively switch environments. I assume that can't be the only way to use Python version executables inside of a virtual environment, as that seems like that would remove the point of having a separate pyenv activate CLI API for pyenv-virtualenv.

Solution

You can directly execute the pipx binary in the pyenv prefix, it should work correctly.

Pyenv's shims mechanism isn't really designed for global binaries like this. I naively expected that the global environment would act as fallback when a local environment doesn't have an installed binary, but I think pyenv only looks at the system Python before falling back to $PATH.

So, if you don't install pipx into system (which, if you're not installing a system pip, I doubt you're doing), then the naive fallback doesn't work. An alternative is to run pyenv with a temporary environment, i.e. PYENV_VERSION=my-pipx-env pyenv exec pipx.

I'd want to make this an executable, so I'd suggest adding something like this into a special PATH directory that takes precedence from the pyenv paths:

#!/usr/bin/env bash
set -eu

export PYENV_VERSION="pipx"
exec "${PYENV_ROOT}/libexec/pyenv" exec pipx "$@"

Although, I'd be tempted to forgo the whole activation logic and just directly exec the pipx binary from the environment /bin to avoid running into any shell configuration errors.