Python PEX loading

Question

I've been trying to wrap my head around the python pex utility (https://pex.readthedocs.org/en/latest/) for bundling some applications into .pex files for deployment.

My app isn't large enough to require twitters pants build tool, as well as I have some build requirements which pants doesn't address. I did however, try the pants tools build system using python_binary which resulted in pex files with the sources loaded into the pex files root. The BUILD files in pants accept a sources property for python_binary which can be a glob for files within the directory the build is running in, however, pants is using the pex programmatic API and not the command-line utility.

The problem is when I use the pex command-line utility on it's own, it seems to want distributions (i.e. folders set up with a setup.py, etc...) and wants to install my code into the .deps folder in the pex file rather than just copying the python files into the root of the pex file like pants seems to do.

Is copying the files over (not installing a package) not possible through the command-line pex tool?

Answer 1

As of pex 1.0.0 there is no facility to glob up files and directories directly, you must have a setup.py as you suggest, or use pants (no longer Twitter's by the way - independent).

So you have 3 paths forward (#1 you already know, but spelling it out for others):

1. Create a setup.py and point the pex tool at its dir

   $ tree -h
   .
   ├── [4.0K]  lib
   │   ├── [   0]  __init__.py
   │   ├── [  38]  lib.py
   │   └── [  68]  main.py
   └── [  76]  setup.py
   
   1 directory, 4 files
   $ cat lib/lib.py 
   def func():
       return 'func in lib'
   
   $ cat lib/main.py 
   from .lib import func
   
   if __name__ == '__main__':
       print(func())
   
   $ cat setup.py 
   from setuptools import setup
   
   setup(
       name='lib',
       packages=['lib']
   )
   
   $ pex . -e lib.main -o lib.pex
   $ ./lib.pex 
   func in lib
   

   NB: The . in the pex command line is the bit pointing pex at this dir's setup.py

2. File an issue against pex to support a set of files in place of a requirement / setup.py. You can do that here.

3. File issues against pants to support the requirements you have that it does not address. You can do that here

As a pants committer I can say that we're working towards making pants easier and easier to use such that no project is too small. You should be able to pip install pantsbuild.pants.backend.python && touch pants.ini and be up and running in a python-only repo but we're not there today.