I have a Python project which I would like to distribute as a Pex or shiv self-contained Python-executable package, in the spirit of the Python Packaging Guide, "Depending on a pre-installed Python" section. My project is structured in the spirit of PEP518, and it has a pyproject.toml file. My project also includes a few libraries not in the Python Standard Library, so I use pipenv to manage those.
How to I build the pex package using a backend which I can specify in the [build-backend] of my pyproject.toml file?
The documentation for pex and shiv show how to build self-contained packages from the command line, or via setuptools.py, but not using the PEP518 structure and pyproject.toml. At least, not as far as I have been able to discover. (And, by "self-contained", I mean all Python language packages, but I am happy to use an existing Python 3 interpreter on the destination system.)
Note that of the three executable packages listed in the Packaging Guide, zipapps does not seem like a fit for me. It doesn't give me a way to manage my external libraries.
Update: some specific invocations, per request.
I currently use build as my build frontend. I use setuptools as my build backend. My pyproject.toml file currently reads,
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
I currently build a wheel via this shell command:
(MyPipenvVenv) % python -m build
…[many lines of output elided]…
Successfully built MyProject-0.0.6a0.tar.gz and MyProject-0.0.6a0-py3-none-any.whl
I can build a self-contained app (which relies on the system's Python interpreter) using these pipenv and shiv commands:
(MyPipenvVenv) % pipenv requirements > requirements.txt
(MyPipenvVenv) % shiv --console-script myapp -o app/myappfile.pyz -r requirements.txt .
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Installing backend dependencies: started
Installing backend dependencies: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Collecting click==8.1.3
Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting pip==22.1.2
Using cached pip-22.1.2-py3-none-any.whl (2.1 MB)
Collecting setuptools==62.5.0
Using cached setuptools-62.5.0-py3-none-any.whl (1.2 MB)
Collecting shiv==1.0.1
Downloading shiv-1.0.1-py2.py3-none-any.whl (19 kB)
Building wheels for collected packages: MyProject
Building wheel for MyProject (pyproject.toml): started
Building wheel for MyProject (pyproject.toml): finished with status 'done'
Created wheel for MyProject: filename=MyProject-0.0.6a0-py3-none-any.whl size=5317 sha256=bbcc…cf
Stored in directory: /private/var/folders/…/pip-ephem-wheel-cache-eak1xqjp/wheels/…cc1d
Successfully built MyProject
Installing collected packages: MyProject, setuptools, pip, click, shiv
Successfully installed MyProject-0.0.6a0 click-8.1.3 pip-22.1.2 setuptools-62.5.0 shiv-1.0.1
What I want is to give the command to the PEP 517 front-end, have the pyproject.toml specify that the resulting build work be done by shiv, and point to whatever configuration shiv needs. I want the result be a self-contained app file app/myappfile.pyz. e.g.
(MyPipenvVenv) % python -m build
…[many lines of output elided]…
Successfully built MyProject
Installing collected packages: MyProject, setuptools, pip, click, shiv
Successfully installed MyProject-0.0.6a0 click-8.1.3 pip-22.1.2 setuptools-62.5.0 shiv-1.0.1
My pyproject.toml file would be something like,
[build-system]
requires = ["shiv"]
build-backend = "shiv.build_something_something"
As far as I know, shiv is not a "PEP 517 build back-end" (neither is pex), so it is not possible to write something like the following in pyproject.toml:
[build-system]
requires = ["shiv"]
build-backend = "shiv.build_something_something"
As discussed there, the PEP 517 interface is targeted at the generation of source distributions (sdist) and wheels only.
From my point of view, I consider tools like shiv and pex that generate zipapps to be (at least) one layer above. And when working at this level, it does not matter whether or not sdists and/or wheels are generated via the PEP 517 interface, in other words it does not matter whether or not pyproject.toml files are involved. I assume that shiv and pex either consume wheels and sdists that are already available (maybe downloaded from PyPI) or they delegate the "build" step to a 3rd party tool (maybe pip, maybe build), I do not know and it does not matter.
From my point of view, the input that makes the most sense to get a zipapp as output is some kind of "lock file", and not a (PEP 517) pyproject.toml file. Zipapps are basically one whole "virtual environment" in a single file. It means that the Python interpreter is fixed, and each dependency (direct or indirect) is fixed. This is best described with a lock file.
The requirements.txt files while not strictly lock files, are probably what is the closest thing with enough availability and support in the Python packaging ecosystem. And as far as I know the requirements.txt files are the only "lock file"-ish format that tools like shiv and pex accept as input.
So my recommendation for you would be to focus on requirements.txt files to provide as input to pex or shiv. As you are already doing.
In the Python packaging ecosystem...
It looks like PDM has a real lock file format and already has support for generating zipapps via a plugin pdm-packer.
Poetry also has a lock-file format and they are somewhat looking into supporting zipapps as well
There are discussions and work going on towards a standardized lock file format. But it is difficult work, and will probably still take some time to reach a conclusion.