I need to run certain commands after building a Cython module with python setup.py bdist_wheel or an equivalent command. Is this possible?
Specifically here is the situation:
I am working on a Cython program for osx. It links to certain .dylib files in my home directory and to get the program to build, you either have to run
export DYLD_LIBRARY_PATH=/path/to/parent/dir
(which is a hassle) or include a directive in setup.py to run
install_name_tool -change @executable_path/my.dylib /path/to/my.dylib
What this basically does is it tells the binary where to look for the dylib. Neither option is ideal, but the latter seems slightly more portable. Right now I just added these instructions to the bottom of setup.py, but this is still not portable: let's say I package and upload my package as follows:
python setup.py sdist # I think this collects source files into a .tar
python setup.py bdist_wheel # I think this builds a binary. This is where I assume the post-compilation stuff should happen.
twine upload dist/* # This puts stuff up on PyPi.
Now, I have read the documentation and a few tutorials, and I am admittedly still not 100% sure what these commands do. But I know that if subsequently, in another project, I run
pip install mypackage
The dylib issue rears its ugly head:
ImportError: dlopen(/path/to/my/module.cpython-36m-darwin.so, 2):
Library not loaded: @executable_path/my.dylib
Referenced from: /path/to/my/module.cpython-36m-darwin.so
Reason: image not found
The solution it seems is to add some kind of automatic post-compilation direction to setup.py to indicate that the install_name_tool operation needs to be performed. Assuming this is possible, how would one do it? Ideally I would like to use off-the-shelf Cython.Build.cythonize and setuptools.Extension, but maybe I need to do something more customized. Thanks for your suggestions!
Danny correctly pointed out that running install_name_tool is not a portable solution and instead the dylib should somehow be included in the package. delocate is the tool for the job, but the first challenge that I encountered was that delocate doesn't work with @executable_path:
❯ delocate-wheel -w fixed_wheels -v dist/*.whl
Fixing: dist/my-package-1.0.9-cp36-cp36m-macosx_10_12_x86_64.whl
/Users/ethan/virtualenvs/my-package/lib/python3.6/site-packages/delocate/delocating.py:71: UserWarning: Not processing required path @executable_path/my.dylib because it begins with @
'begins with @'.format(required))
Using otool, I was able to verify that the id of my.dylib uses @executable_path:
❯ otool -L my.dylib
my.dylib:
@executable_path/my.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1258.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
To get rid of @executable_path I ran the following:
❯ install_name_tool -id my.dylib my.dylib
Now look at the output of otool -L:
❯ otool -L my.dylib
my.dylib:
my.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1258.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
In general, this is not a good id to use for a dylib, but it is enough for delocate-wheel to work with. I rebuilt my .whl:
python setup.py bdist_wheel
And now when I ran delocate_wheel:
❯ delocate-wheel -w fixed_wheels -v dist/*.whl
Fixing: dist/my.whl
Traceback (most recent call last):
...
delocate.delocating.DelocationError: library "/Users/ethan/my-package/my.dylib" does not exist
This clearly tells us where delocate expects to find my.dylib. So I copied my.dylib to /Users/ethan/my-package/my.dylib and reran the command:
❯ delocate-wheel -w fixed_wheels -v dist/*.whl
Fixing: dist/my.whl
Copied to package .dylibs directory:
/Users/ethan/my-package/my.dylib
Success! What is this .dylibs directory? I ran tar -xvf my.whl to unpack the wheel and examine its contents:
❯ tree -a fixed_wheels/my-package
fixed_wheels/my-package
├── .dylibs
│ └── my.dylib
├── __init__.py
└── package.cpython-36m-darwin.so
As you can see, the my.dylib has been copied into a .dylibs/ directory that gets packaged in the my.whl. After uploading my.whl to pypi, I was able to download and run the code just fine.
Have a look at delocate and the Linux equivalent auditwheel.
Rather than having to change link loader paths at runtime, the delocate tool embeds third party libraries into a wheel and adjusts load time paths accordingly.
The result is a distributable binary wheel with all dependent libraries correctly included.
Setting runtime load paths is not portable and will not work across python versions or machine architectures.
For example for the above, after a wheel is created by python setup.py bdist_wheel run:
delocate-wheel -w fixed_wheels <wheel file>
A fixed wheel file will be placed under fixed_wheels directory. Run with -v to see what libraries it has embedded.