Call a function from a cython generated .so file inside a c++ code

Question

My goal is to call python functions from C++. These python function must be compiled with cython in a .so file. The .so file must be the one that communicate with the c++ program.

Before all:

I am on Ubuntu, I am working with miniconda with python3.9. I am in a folder (~/exp) made like this:

• exp
  • exp
    • __ init __.py
    • main.py
• setup.py
• run.cpp
• run.py

I translate the main.py to main.pyx, the file contains this code:

def add(a, b):
    return a+b

def entry_point():
    print(add(21,21))

I compiled this script with cython and obtained a .so file, with this setup.py script:

from setuptools import setup
from Cython.Build import cythonize

setup(
   name='exp',
   ext_modules=cythonize("exp/main.pyx"),
   libraries=[('python3.9', {'include_dirs': ["~/miniconda3/include/python3.9"]})],
   library_dirs=['~/miniconda3/lib']
)

And this command:

python3 setup.py build_ext --inplace

Now I have a main.cpython-39-x86_64-linux-gnu.so file in ~/exp/exp.

When I launch (run.py) which contains:

from exp.main import entry_point

if __name__ == "__main__":
    entry_point()

I have a normal behavior : It returns 42.

Now, here come the problems

I compile my run.cpp source, which contains :

#include <iostream>
#include <dlfcn.h>
#include "Python.h"


int main(int argc, char *argv[]) {
   setenv("PYTHONPATH",".",1);
   Py_Initialize();
   PyObject *pName, *pModule, *pDict, *pFunc, *pValue, *presult;
   // Initialize the Python Interpreter
   Py_Initialize();

   // Build the name object
   pName = PyUnicode_FromString((char*)"exp/main");

   // Load the module object
   pModule = PyImport_Import(pName);


   // pDict is a borrowed reference 
   pDict = PyModule_GetDict(pModule);


   // pFunc is also a borrowed reference 
   pFunc = PyDict_GetItemString(pDict, (char*)"entry_point");

   Py_DECREF(pValue);

   // Clean up
   Py_DECREF(pModule);
   Py_DECREF(pName);
   Py_Finalize();
}

with the command :

g++ -Wall -I~/miniconda3/include/python3.9 run.cpp -L~/miniconda3/lib -lpython3.9 -o run.o -ldl

And then execute : ./run.o To end with a beautiful :

Segmentation fault (core dumped)

I tried with dlopen without success either. Maybe I miss something, any help would be welcome.

Thank you :)

Answer 1

Firstly, thank you to Craig Estey and DavidW for their comments.

So I finally was able to make it work, two things was wrong:

1. pValue was not used, so the Py_DECREF raised an Error
2. the module path "exp/main" was indeed not valid, but "exp.main" was valid.

A very last thing. Something I omitted was the PyObject_CallObject that allows to call my PyObject pFunc.

I've finally got my '42' answer.

Here the final code:

#include <iostream>
#include <dlfcn.h>
#include "Python.h"


int main(int argc, char *argv[]) {
   setenv("PYTHONPATH",".",1);
   Py_Initialize();

   PyObject *pName, *pModule, *pDict, *pFunc, *presult;
   // Initialize the Python Interpreter

   // Build the name object
   pName = PyUnicode_FromString((char*)"exp.main");

   // Load the module object
   pModule = PyImport_Import(pName);


   // pDict is a borrowed reference 
   pDict = PyModule_GetDict(pModule);


   // pFunc is also a borrowed reference 
   pFunc = PyDict_GetItemString(pDict, (char*)"entry_point");

   presult = PyObject_CallObject(pFunc, NULL);

   // Py_DECREF(pValue);

   // Clean up
   Py_DECREF(pModule);
   Py_DECREF(pName);
   Py_Finalize();
}

(Craig pointed out that executable file might not finish by '.o', learn more: What is *.o file) So, the new compile command is:

g++ -Wall -I~/miniconda3/include/python3.9 run.cpp -L~/miniconda3/lib -lpython3.9 -o run