Dynamic loading of Python C extension fails due to missing symbols

Question

I am in the situation that I have an executable (main.c) that dynamically loads a shared object (py_plugin.c) which is in turn is linked to python.

However, when the python plugin attempts to import a module whose dependencies are not linked to libpython I get the following error:

ImportError: /usr/lib/python2.7/lib-dynload/bz2.x86_64-linux-gnu.so: undefined symbol: PyExc_SystemError

As far as I can deduce this implies that the library bz2.x86_64-linux-gnu.so does not have access to the python symbols.

Note that the error is specific to the "bz2" package because I am forcing it to surface using the minimal working example at the end of the question. There I am performing an explicit import of the "bz2", which loads the library bz2.x86_64-linux-gnu.so, inside the python plugin (py_plugin.c).

Looking at the dependencies I verify that:

1. The library bz2.x86_64-linux-gnu.so is not linked to python

       usr@cmptr $ ldd /usr/lib/python2.7/lib-dynload/bz2.x86_64-linux-gnu.so
       linux-vdso.so.1 =>  (0x00007ffd511fb000)
       libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8c63a0a000)
       libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8c6362a000)
       libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f8c6341a000)
       /lib64/ld-linux-x86-64.so.2 (0x00007f8c63e33000)
   

2. But my python plugin is:

       usr@cmptr $ ldd py_plugin.so
       linux-vdso.so.1 =>  (0x00007ffc1ef5c000)
       libpython2.7.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 (0x00007f56ac01c000)
       libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f56abc3c000)
       libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f56aba1d000)
       libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f56ab800000)
       libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f56ab5fc000)
       libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f56ab3f8000)
       libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f56ab0a2000)
       /lib64/ld-linux-x86-64.so.2 (0x00007f56ac79a000)
   

My questions are the following:

1. The problem itself is quite obvious, but why is the python symbols not available when my plugin clearly is linked to libpython?

2. Does anyone have an idea for how to resolve this issue when I can not link python to the main executable (it is a pre-compiled binary)?

From this e-mail thread I understand that the source of the error may be because differences in the distribution philosophy of python (I am running a Ubuntu-based distro). This bug report also highlights the issue.

---

Example producing error

generate libs/execs

gcc $(pkg-config --cflags python) -shared -o py_plugin.so py_plugin.c $(pkg-config --libs python)
gcc -o main main.c -lltdl

The putput from pkg-config on my system is:

pkg-config --cflags python
-I/usr/include/python2.7 -I/usr/include/x86_64-linux-gnu/python2.7

pkg-config --libs python
-lpython

file: main.c

#include <ltdl.h>
#include <stdio.h>
typedef int(*dyn_fptr)();

int main()
{
  if(lt_dlinit()) {
    return -1;
  }

  lt_dlhandle handle = lt_dlopen("./py_plugin.so");

  dyn_fptr func = (dyn_fptr)lt_dlsym(handle, "func");
  int a = func(); // <----------------------------------- Call "func" in py_plugin

  lt_dlclose(handle);
  return 0;
}

file: py_plugin.c

#include <Python.h>

int func()
{

  Py_Initialize();
  PyObject *pName = PyString_FromString("bz2");

  PyObject *pModule = PyImport_Import(pName); // <--------------------- ERROR
  Py_DECREF(pName);

  if(!pModule) {
    PyErr_Print();
  }

  Py_Finalize();

  return 0;
}

Answer 1

Closing the thread for future reference.

As was pointed out in the comment to my question the problem here is that Python is (currently) packaged differently on different Linux distributions.

In order to fix the issue one must ensure that the Python symbols are visible to libraries that are loaded within the python plugin.

As far as I can tell there are three ways of dealing with it:

1. Link the main executable to Python.
2. Set the RTDL_GLOBAL option (only possible when using dlopen, dlopen("lib.so", RTDL_NOW | RTDL_LAZY | RTDL_GLOBAL), and is not guaranteed to be supported on all systems).
3. Set the environment variable LD_PRELOAD, export LD_PRELOAD="/path/to/libpython2.7.so.1.0", to enforce certain libraries to be loaded before any other library. For me this worked for my little example plugin, but caused harmful interactions when dealing with a larger software framework.