Search code examples
pythonshared-librariesctypesdynamic-linking

ctypes not loading share library from the standard paths of dlopen

I have a compiled .so file in linux that links to openblas under my LD_LIBRARY_PATH, and this is confirmed by ldd:

libopenblas.so.0 => /home/username/.local/lib/libopenblas.so.0 (0x000014a262823000)

However, when loading this in the anacodna python, this version is not linked and the one in the anaconda lib is used instead.

This is seen by the following error (I use this error message to print out the true openblas file that is linked):

In [1]: import ctypes
   ...: ctypes.cdll.LoadLibrary('libopenblas.so.0').name
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-40ebc71b2d6c> in <module>
      1 import ctypes
----> 2 ctypes.cdll.LoadLibrary('libopenblas.so.0').name

/etc/anaconda3/lib/python3.7/ctypes/__init__.py in __getattr__(self, name)
    375         if name.startswith('__') and name.endswith('__'):
    376             raise AttributeError(name)
--> 377         func = self.__getitem__(name)
    378         setattr(self, name, func)
    379         return func

/etc/anaconda3/lib/python3.7/ctypes/__init__.py in __getitem__(self, name_or_ordinal)
    380
    381     def __getitem__(self, name_or_ordinal):
--> 382         func = self._FuncPtr((name_or_ordinal, self))
    383         if not isinstance(name_or_ordinal, int):
    384             func.__name__ = name_or_ordinal

AttributeError: /etc/anaconda3/lib/python3.7/lib-dynload/../../libopenblas.so.0: undefined symbol: name

/etc/anaconda3/lib/ is not on my LD_LIBRARY_PATH, per the documentations of dlopen, this should not be found. If I'm not allowed to modify the content in /etc/anaconda3/lib/, is there away to set higher priority to LD_LIBRARY_PATH and have ctypes finding my customized build of openblas?

Solution

  • It seems that python is compiled with RPATH pointing to $ORIGIN/../../lib, which always has higher precedence than LD_LIBRARY_PATH. To override this, I compile the library with -Wl,-rpath,--enable-new-dtags flags to set an empty RUNPATH which will replace RPATH but with lower precedence than LD_LIBRARY_PATH. This fixed my problem and allows my library to link to libopenblas.so.0 outside the anaconda lib.