I am writing the C++ and Python side of a library that exposes some functionality in our software written in C++ to Python scripts. I'm compiling some source files of interest and a wrapper file that looks like below into a shared library and loading that library using ctypes.
extern "C" {
PyObject *py_get_cxx_set_EXAMPLE(void)
{
std::set<long> cset = get_cxx_set_for_python();
PyGILState_STATE gstate = PyGILState_Ensure();
PyObject *pyset = PySet_New(NULL);
for (long c_long: cset)
PySet_Add(pyset, PyLong_FromLong(c_long));
PyGILState_Release(gstate);
return pyset;
}
}
and on the python side:
example_lib.py_get_cxx_set_EXAMPLE.restype = ctypes.py_object
for i in range(0, 1000):
ret = example_lib.py_get_cxx_set_EXAMPLE()
I find that the first few calls would be successful, but the C++ code would segfault in the middle of the loop. Upon GDB'ing, I would find the end of the callstack like this:
#0 0x000055555563244a in PyErr_Occurred ()
#1 0x000055555562a387 in _PyObject_GC_Malloc ()
#2 0x0000555555629ebd in _PyObject_GC_New ()
#3 0x000055555562b23c in PyDict_New ()
#4 0x00007ffff66df9be in python::to_python_object<db::pmbus_diagnostics> (t=...) at python_wrapper/python.hpp:101
It looks like the Python runtime refuses to make more Python objects (in this case, a dict) for me...!
What did I do wrong in the C++ code?
EDIT:: Updated, see answer
OK, I forgot to add the code to acquire and release the global interpreter lock for some class of functions. Sorry for the silly question.
Have faith in Python kids.