Search code examples
pythonc++python-2.7python-c-api

Returning new PyObject * from C++ to Python eventually segfaults


I am writing the C++ and Python side of a library that exposes some functionality in our software written in C++ to Python scripts. I'm compiling some source files of interest and a wrapper file that looks like below into a shared library and loading that library using ctypes.

extern "C" {
    PyObject *py_get_cxx_set_EXAMPLE(void)
    {
        std::set<long> cset = get_cxx_set_for_python();
        PyGILState_STATE gstate = PyGILState_Ensure();
        PyObject *pyset = PySet_New(NULL);
        for (long c_long: cset)
            PySet_Add(pyset, PyLong_FromLong(c_long));
        PyGILState_Release(gstate);
        return pyset;
    }
}

and on the python side:

example_lib.py_get_cxx_set_EXAMPLE.restype = ctypes.py_object
for i in range(0, 1000):
    ret = example_lib.py_get_cxx_set_EXAMPLE()

I find that the first few calls would be successful, but the C++ code would segfault in the middle of the loop. Upon GDB'ing, I would find the end of the callstack like this:

#0  0x000055555563244a in PyErr_Occurred ()
#1  0x000055555562a387 in _PyObject_GC_Malloc ()
#2  0x0000555555629ebd in _PyObject_GC_New ()
#3  0x000055555562b23c in PyDict_New ()
#4  0x00007ffff66df9be in python::to_python_object<db::pmbus_diagnostics> (t=...) at python_wrapper/python.hpp:101

It looks like the Python runtime refuses to make more Python objects (in this case, a dict) for me...!

What did I do wrong in the C++ code?

EDIT:: Updated, see answer


Solution

  • OK, I forgot to add the code to acquire and release the global interpreter lock for some class of functions. Sorry for the silly question.

    Have faith in Python kids.