Search code examples
pythoncpsutilc-api

why does python import causes an error with PyImport_ImportModule but not with python interpreter

I have a python module mymodule that contains an import psutil statement. I can run it using python interpreter without any errors.
However, I fail to load it with PyImport_ImportModule("mymodule") (from python C API), due to an undefined symbol in import psutil.
I don't understand what causes this, and since the C code is part of a shared object that is aimed to load various python modules, I will appreciate any help.

I'm suspecting that this is a bigger issue than this specific module, but here are some specific details-

traceback

Traceback (most recent call last):
  File "/path/to/mymodule.py", line 7, in <module>
    import psutil
  File "/usr/local/lib/python3.8/dist-packages/psutil/__init__.py", line 99, in <module>
    from . import _pslinux as _psplatform
  File "/usr/local/lib/python3.8/dist-packages/psutil/_pslinux.py", line 26, in <module>
    from . import _psutil_linux as cext
ImportError: /usr/local/lib/python3.8/dist-packages/psutil/_psutil_linux.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyExc_RuntimeError

ldd

linux-vdso.so.1 (0x00007ffe91b5e000)
        libpython3.8.so.1.0 => /lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f5a1b41a000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5a1b3f7000)
        libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f5a1b121000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5a1af3f000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5a1af24000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5a1ad30000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f5a1ad02000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5a1ace6000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5a1ace0000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f5a1acdb000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5a1ab8c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5a1b9a8000)

psutil on python interpreter:

>>> import psutil
>>> psutil.__file__
'/usr/local/lib/python3.8/dist-packages/psutil/__init__.py'
Solution

  • OK, so the issue here was the way dlopen link the libraries.
    For my case, updating dlopen from
    dlopen(so_lib_name, RTLD_NOW)
    to
    dlopen(so_lib_name, RTLD_NOW|RTLD_GLOBAL)
    worked!