TLDR: Pants fetches OS X specific wheels bc I'm developing on Mac. How can I avoid this, or specify that I will deploy to Ubuntu?
Full story:
Trying to package a Python application with Pants. Going great so far, but ran into a problem which I've been stuck at for a while. I'm developing on a macbook but deploying to EC2 Ubuntu.
Here's what I've done so far:
./pants run.py backend:admin_server which runs fine and generated dist/admin_server.pexHowever when I run the application there, I get:
Failed to execute PEX file, missing compatible dependencies for:
mysql-python
pycrypto
The problem seems to be that Pants takes OS X specific wheels for these 2:
pex: - MySQL_python-1.2.5-cp27-none-macosx_10_11_intel.whl pex: - pycrypto-2.6.1-cp27-none-macosx_10_11_intel.whl
How can I avoid that, or specify which OS they should run on?
Here's the full output:
ubuntu@ip-***:~$ export PEX_VERBOSE=1
ubuntu@ip-***:~$ python admin_server.pex
pex: Found site-library: /usr/local/lib/python2.7/dist-packages
pex: Found site-library: /usr/lib/python2.7/dist-packages
pex: Tainted path element: /usr/local/lib/python2.7/dist-packages
pex: Tainted path element: /usr/lib/python2.7/dist-packages
pex: Scrubbing from site-packages: /usr/local/lib/python2.7/dist-packages
pex: Scrubbing from site-packages: /usr/lib/python2.7/dist-packages
pex: Scrubbing from user site: /home/ubuntu/.local/lib/python2.7/site-packages
pex: Failed to resolve a requirement: MySQL-python==1.2.5
pex: Failed to resolve a requirement: pycrypto==2.6.1
pex: Unresolved requirements:
pex: - mysql-python
pex: - pycrypto
pex: Distributions contained within this pex:
pex: - six-1.10.0-py2.py3-none-any.whl
pex: - protobuf-2.6.1-py2.7.egg
pex: - setuptools-19.5-py2.py3-none-any.whl
pex: - MySQL_python-1.2.5-cp27-none-macosx_10_11_intel.whl
pex: - pycrypto-2.6.1-cp27-none-macosx_10_11_intel.whl
pex: - futures-3.0.4-py2-none-any.whl
pex: - webapp2-2.5.2-py2-none-any.whl
pex: - requests-2.9.0-py2.py3-none-any.whl
pex: - jmespath-0.9.0-py2.py3-none-any.whl
pex: - beautifulsoup4-4.4.1-py2-none-any.whl
pex: - python_dateutil-2.4.2-py2.py3-none-any.whl
pex: - boto3-1.2.3-py2.py3-none-any.whl
pex: - WebOb-1.5.1-py2.py3-none-any.whl
pex: - cssutils-1.0.1-py2-none-any.whl
pex: - webapp2_static-0.1-py2-none-any.whl
pex: - Paste-2.0.2-py2-none-any.whl
pex: - docutils-0.12-py2-none-any.whl
pex: - botocore-1.3.22-py2.py3-none-any.whl
pex: - protobuf_to_dict-0.1.0-py2-none-any.whl
Failed to execute PEX file, missing compatible dependencies for:
mysql-python
pycrypto
PS: to make sure I didn't include my versions of the python libraries, I pip uninstalled both PyCrypto and MySQL-Python.
One of the nice things about distributing your project as a PEX file is that you can prepare it to run on multiple platforms. For example, one PEX can run on both Linux and Mac platforms. For many projects, there is nothing special to do other than build a PEX. But when your project has dependencies on platform specific binary code, you will need to perform some extra steps.
One example of a library that contains platform specific code is the psutil library. It contains C code that it compiled into a shared library when the module is installed. To create a PEX file that contains such dependencies, you must first provide a pre-built version of that library for all platforms other than the one where you are running pants.
The easiest way to pre-build libraries is to use the pip tool to build wheels.
This recipe assumes the following:
Let’s take a simple program that references a library and create a pex from it.
# src/python/ps_example/main.py
import psutil
for proc in psutil.process_iter():
try:
pinfo = proc.as_dict(attrs=['pid', 'name'])
except psutil.NoSuchProcess:
pass
else:
print(pinfo)
With Pants, you can define an executable by defining a python_binary target in a BUILD file:
# src/python/ps_example/BUILD
python_binary(name='ps_example',
source = 'main.py',
dependencies = [
':psutil', # defined in requirements.txt
],
)
# Defines targets from specifications in requirements.txt
python_requirements()
In the same directory, list the python libraries in a requirements.txt file:
# src/python/ps_example/requirements.txt
psutil==3.1.1
Now, to to make the multi-platform pex, you'll need access to a Linux box to create the linux version of psutil wheel. Copy the requirements.txt file to the linux machine, then, execute the pip tool:
linux $ mkdir ~/src/cookbook/wheelhouse
linux $ pip wheel -r src/python/multi-platform/requirements.txt \
--wheel-dir=~/src/cookbook/wheelhouse
This will create a platform specific wheel file.
linux $ ls ~/src/cookbook/wheelhouse/
psutil-3.1.1-cp27-none-linux_x86_64.whl
Now, you will need to copy the platform specific wheel over to the machine where you want to build your multi-platform pex (in this case, your mac laptop). If you use this recipe on a regular basis, you will probably want to configure a Python Respository to store your pre-built libraries.
We’ll use the same BUILD file setup as in above, but modify python_binary to specify the platforms= parameter.
# src/python/ps_example/BUILD
python_binary(name='ps_example',
source = 'main.py',
dependencies = [
':psutil', # defined in requirements.txt
],
platforms=[
'linux-x86_64',
'macosx-10.7-x86_64',
],
)
# Defines targets from specifications in requirements.txt
python_requirements()
You will also need to tell pants where to find the pre-built python packages. Edit pants.ini and add:
[python-repos]
repos: [
"%(buildroot)s/wheelhouse/"
]
Now, copy the file psutil-3.1.1-cp27-none-linux_x86_64.whl over to the mac workstation and place it in a directory named wheelhouse/ under the root of your repo.
Once this is done you can now build the multi-platform pex with
mac $ ./pants binary src/python/py_example
You can verify that libraries for both mac and Linux are included in the pex by unzipping it:
mac $ unzip -l dist/ps_example.pex | grep psutil
17290 12-21-15 22:09 .deps/psutil-3.1.1-cp27-none-linux_x86_64.whl/psutil-3.1.1.dist-info/DESCRIPTION.rst
19671 12-21-15 22:09 .deps/psutil-3.1.1-cp27-none-linux_x86_64.whl/psutil-3.1.1.dist-info/METADATA
1340 12-21-15 22:09 .deps/psutil-3.1.1-cp27-none-linux_x86_64.whl/psutil-3.1.1.dist-info/RECORD
103 12-21-15 22:09
... .deps/psutil-3.1.1-cp27-none-macosx_10_11_intel.whl/psutil-3.1.1.dist-info/DESCRIPTION.rst
19671 12-21-15 22:09 .deps/psutil-3.1.1-cp27-none-macosx_10_11_intel.whl/psutil-3.1.1.dist-info/METADATA
1338 12-21-15 22:09 .deps/psutil-3.1.1-cp27-none-macosx_10_11_intel.whl/psutil-3.1.1.dist-info/RECORD
109 12-21-15 22:09
...