Search code examples
pythonshared-librariessetuptools

Package shared object (library) with Python Egg / Wheel

I've done this so far:

  1. Created MANIFEST.in with: include path/to/libfoo.so
  2. Created setup.py that after calling setupt.py install puts the libfoo.so into /usr/local/lib/python/site-packages/foo.egg/path/to/libfoo.so.

This, of course, doesn't help Python to find libfoo when it's required at run time. What do I need to do to make Python actually find this library?

Note

This library doesn't have Python bindings, it's just a shared library with some native code in it. It is called from another shared library which sits in /usr/local/lib/python/site-packages/foo.egg/path/wrapped.cpython-36m-x86_64-linux-gnu.so.

Solution

  • If you want to hard-code the location of shared library, you can use the rpath option. For that you would do something like..

    python setup.py build_ext --rpath=/usr/local/lib/python/site-packages/foo.egg/path/to

    Where the setup.py above is the script used to build wrapped.cpython-36m-x86_64-linux-gnu.so and the rpath is the path to libfoo.so Of course you should be able to put this directly inside the build script, depending on what that process looks like.

    -rpath=dir

    Add a directory to the runtime library search path. This is used when linking an ELF executable with shared objects. All -rpath arguments are concatenated and passed to the runtime linker, which uses them to locate shared objects at runtime. The -rpath option is also used when locating shared objects which are needed by shared objects explicitly included in the link

    If it's not an option to update the build process for wrapped.cpython-36m-x86_64-linux-gnu.so I think your only option it to put libfoo.so somewhere that's in the load library path or manually add the location at run-time.

    In answer to a few of your follow-on questions...

    The system load library locations come from /etc/ld.so.conf and references the locations in the ld.so.conf.d directory. The ldconfig command rebuilds the the cache for shared libraries from this data so if you change things be sure to call this command.

    At the command line or in your .bashrc you can use export LD_LIBRARY_PATH=.... to add additional directories to the search path.

    You can manually load shared objects. See https://docs.python.org/2/library/ctypes.html Loading shared libraries.

    I haven't tried this myself but I've read that if you manually load a subordinate shared library in your python code and then import the higher level library, the linker won't have to go out and find the lower one since it's already loaded. This would look something like...

    import ctypes
foolib = ctypes.CDLL('/full/path/to/libfoo.so')
import wrapped

    There's a number of examples on StackOverflow on how to do this and lots of additional info/examples on manipulating the library search paths.