Search code examples
pythonctypesblasintel-mklpardiso

Calling Pardiso 6 in Python

I'm trying to use Pardiso 6 sparse solver library in Python. The problem is that I can't seem to load the Pardiso shared object (SO). Here's the error that I get when calling

import ctypes
pardiso = ctypes.CDLL(pardiso_so_address)

Traceback (most recent call last):
  File "test.py", line 27, in <module>
    pardiso = ctypes.CDLL(lib720)
  File "/home/amin/anaconda3/envs/idp/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: ./libpardiso600-GNU720-X86-64.so: undefined symbol: sgetrf_

I'd really appreciate it if someone could shed some light on this.

PS. I already contacted Pardiso developers and they told me that I need to link against optimized BLAS, but I already have MKL installed via conda.

Update 1: I installed mkl via conda, but it didn't help. Strangely, I added import scipy to the header and the error went away. The same thing happens if I add import mkl. So, for some reason, unless scipy or mkl are manually imported, the .so doesn't know that a lapack installation exists. Anyway, now another error is thrown, which I think might be related the libgfortran library. Here's the error

Traceback (most recent call last):
  File "test.py", line 34, in <module>
    pardiso = ctypes.CDLL(lib720)
  File "/home/amin/anaconda3/envs/test/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: ./libpardiso600-GNU720-X86-64.so: undefined symbol: _gfortran_st_close

I double-checked to see if libgfortran is installed, and indeed it is:

(test) PyPardisoProject$ ldconfig -p | grep libgfortran
    libgfortran.so.5 (libc6,x86-64) => /lib/x86_64-linux-gnu/libgfortran.so.5
    libgfortran.so.4 (libc6,x86-64) => /lib/x86_64-linux-gnu/libgfortran.so.4

I think something similar might be at play, i.e. the library is there but it needs to be triggered (similar to what import scipy seems to have done for liblapack, but I have no idea how I can trigger it.

Note: I found an example in C on Pardiso website and tested the .so file against it via

$ gcc pardiso_sym.c -o pardiso_sym -L . -lpardiso600-GNU720-X86-64 -llapack -fopenmp -lgfortran
$ OMP_NUM_THREADS=1 ./pardiso_sym

and it worked with no problem (with the existing libraries on my machine). So, the .so works, it's just that I don't know how to inform it of its dependencies in Python.

Update 2: Here's the output of ldd pardiso_sym:

Scripts$ ldd pardiso_sym
    linux-vdso.so.1 (0x00007ffe7e982000)
    libpardiso600-GNU720-X86-64.so (0x00007f326802d000)
    liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3 (0x00007f3267976000)
    libgfortran.so.4 => /lib/x86_64-linux-gnu/libgfortran.so.4 (0x00007f3267795000)
    libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f326775b000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3267568000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3267545000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f32673f6000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f32685df000)
    libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 (0x00007f3267389000)
    libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007f32670e9000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f32670cf000)
    libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f3267083000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f326707d000)

So, I added the common path, i.e. /lib/x86_64-linux-gnu and /lib64 to PATH and ran the Python script again via:

PATH=$PATH:/lib/x86_64-linux-gnu:/lib64 python padiso_script.py

but the same error is thrown. I also tried adding to LD_LIBRARY_PATH as well, but didn't work either.

Solution

  • The trick is, rather than adding the location of dependencies to system PATHs, you need to explicitly load the dependencies, i.e. lapack, blas, and gfortran in the Python script prior to loading the Pardiso library. Also, it's essential that you explicitly pass the optional mode=ctypes.RLTD_GLOBAL argument to ctypes.CDLL method in order to make the dependencies globally accessible and hence, Pardiso can access them.

    import ctypes
import ctypes.util

shared_libs = ["lapack", "blas", "omp", "gfortran"]
for lib in shared_libs:
    # Fetch the proper name of the dependency
    libname = ctypes.util.find_library(lib)
    # Load the dependency and make it globally accessible
    ctypes.CDLL(libname, mode=ctypes.RTLD_GLOBAL)
# Finally, load the Pardiso library
pardiso = ctypes.CDLL(pardiso_so_address)

    In my experience, if you are inside a conda environment with mkl installed, you only need to list gfortran as dependency and the rest are automatically loaded and accessible, in which case set shared_libs = ["gfortran"].