Embedding python in C++ : strange segmentation faults

Question

(Sorry for the vague title but it shows how dumbfounded I am by the issue).

So I'm running Python code from a C++ program, following the approach described here : https://docs.python.org/2/extending/embedding.html.

Here is the C++ code:

#include <Python.h>
#include <iostream>

int main(int argc, char *argv[])
{
    PyObject *pName, *pModule, *pDict, *pFunc;
    PyObject *pArgs, *pValue;
    int i;

    if (argc < 3) {
        fprintf(stderr,"Usage: call pythonfile funcname [args]\n");
        return 1;
    }

    Py_SetProgramName(argv[0]);
    Py_Initialize();
    PySys_SetArgv(argc, argv); 
    PyObject *sys = PyImport_ImportModule("sys");
    PyObject *path = PyObject_GetAttrString(sys, "path");
    PyList_Append(path, PyString_FromString("."));
    pName = PyString_FromString((char*)argv[1]);
    /* Error checking of pName left out */

    pModule = PyImport_Import(pName);
    Py_DECREF(pName);

    if (pModule != NULL) {
        pFunc = PyObject_GetAttrString(pModule, argv[2]);
        /* pFunc is a new reference */

        if (pFunc && PyCallable_Check(pFunc)) {   
            PyObject *pArgs = PyList_New(4);
            PyList_SetItem(pArgs,0,PyString_FromString("H-SAMPLE1-OH"));
            PyList_SetItem(pArgs,1,PyInt_FromLong(2));
            PyList_SetItem(pArgs,2,PyString_FromString("H-SAMPLE2-OH"));
            PyList_SetItem(pArgs,3,PyInt_FromLong(3));
            PyObject *arglist = Py_BuildValue("(O)", pArgs);
            Py_DECREF(pArgs);

            for(int run = 0; run < 2; run++)
            { 
                std::cout << "begin" << std::endl;

                pValue = PyObject_CallObject(pFunc, arglist);
                //Py_DECREF(arglist);
                if (pValue != NULL) 
                {
                    int py_list_size = PyList_Size(pValue);
                    printf("list size = %d\n",py_list_size);
                    int sub_list_size = 0;
                    for(Py_ssize_t i = 0; i < py_list_size; ++i)
                    {
                        PyObject *pList = PyList_GetItem(pValue, i);
                        sub_list_size = PyList_Size(pList);  
                        if(PyList_Check(pList))
                        {
                            for(Py_ssize_t j = 0; j < sub_list_size; ++j)
                            {
                                PyObject *pListItem = PyList_GetItem(pList, j);
                                double pyNumber = PyFloat_AsDouble(pListItem);
                                std::cout << "pynumber ok" << std::endl;
                                Py_DECREF(pListItem);
                                printf("Result of call: %f\n", pyNumber);
                            }
                        }
                        else
                        {
                            printf("Not list!\n");
                        }  
                        Py_DECREF(pList);
                    }
                    Py_DECREF(pValue);
                }
                else {
                    std::cout << "Else" << std::endl;
                    Py_DECREF(pFunc);
                    Py_DECREF(pModule);
                    PyErr_Print();
                    fprintf(stderr,"Call failed\n");
                    return 1;
                }
            }
        }
        else {
            if (PyErr_Occurred())
                PyErr_Print();
            fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]);
        }
        Py_XDECREF(pFunc);
        Py_DECREF(pModule);
    }
    else {
        PyErr_Print();
        fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);
        return 1;
    }
    Py_Finalize();
    return 0;
}

and here is the toy Python code:

def test(a):
    print a
    return [[1.2,2.6],[4.7,5.6]]

Notice the main loop in the C++ code, iterating over the variable "run". It works like a charm when the code inside the loop is executed only once. If I try to run it more, only twice for example, it goes wrong, I get a segmentation fault. Apparently, the fault happens at line 61, when trying to perform

double pyNumber = PyFloat_AsDouble(pListItem);

I find it really strange. It works fine during the first execution, and then suddenly if doesn't manage to properly get something from pListItem anymore (although it does receive something it recognizes as a list of size 2 and seems to handle all other pyObject pointers correctly). Any idea about what is happening?

To reproduce:

I compiled as follow:

g++ -L/usr/lib/python2.7/config-x86_64-linux-gnu -L/usr/lib -I/usr/include/python2.7 -o ms2pip ms2pip.c -lpthread -ldl  -lutil -lm  -lpython2.7 -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions

And then executed as follow:

$ ./ms2pip python_code test
(so ./executable < python_file_without_py_extension > < function_name >)

Answer 1

I think your problem is that PyList_GetItem() returns a borrowed references. So the problem is with calling Py_DECREF() with both pList and pListItem:

PyObject *pList = PyList_GetItem(pValue, i);
// ...
if(PyList_Check(pList))
{
    for(Py_ssize_t j = 0; j < sub_list_size; ++j)
    {
        PyObject *pListItem = PyList_GetItem(pList, j);
        double pyNumber = PyFloat_AsDouble(pListItem); // <-- Segfault in second iteration after released from first iteration.
        // ...
        Py_DECREF(pListItem); // <-- Bad, released in first iteration.
        // ...
    }
}
//...
Py_DECREF(pList); // <-- Bad, released in first iteration.

pList is a borrowed reference which you are not responsible for releasing with Py_DECREF(). Also, pListItem is also a borrowed reference. So on the first iteration, you release pList as well as each pListItem which is bad. On the second iteration, you grab pList and each pListItem which have all been released and treat them as though they are still stable which is not the case. Because you are accessing freed objects, the program could really fail or give bad results at any of the function calls involving them (e.g., PyList_Size(pList), PyList_GetItem(pList, j), PyFloat_AsDouble(pListItem), Py_DECREF(pListItem), Py_DECREF(pList)).