pybind11: Python to C++ Datatype Conversion Not Working

Question

Problem

I am trying to convert a list of lists, returned by a python function called inside a C++ code. Though the pybind11 library allows type conversion from python data types to C++ data types, my attempt to convert a list of lists returned by python to a std::list of std::list of strings of C++, fails every time.

Code

Here is the python function (The function returns a list of list containing string values):

def return_sheet(self):

     """Returns the sheet in a list of lists

     """

     dataTable = []

     for r in range(self._isheet.nrows):

         datalist = []

         for c in range(self._isheet.ncols):

             datalist.append(self._isheet.cell_value(r,c))

         dataTable.append(datalist)

 return dataTable

And Here I am calling it in C++ using pybind11:

py::list obj = _tool.attr("return_sheet")();

data = py::cast<SheetData>(obj); // This is where the problem lies, This cast crashes the program

Where SheetData is a typedef for:

typedef std::list<std::list<std::string> > SheetData;

While debugging, I found out that the program is actually crashing at this line:

py::object dataTable = _tool.attr("return_sheet")(); // Where _tool.attr("return_sheet")() gives an py::object which is a list of list of str

Does someone know, how can I successfully convert a list of lists of python to std::list of std::list of C++?

EDIT

Here is the python program file I am embedding in c++ [xlanalyser.py] : https://pastebin.com/gARnkMTv

And Here is the c++ code [main.cpp] : https://pastebin.com/wDDUB1s4

Note: All other functions in xlanalyser.py do not cause a crash on embedding in c++ [ only the return_sheet() function causes the crash ]

Answer 1

You can use the Python/C API as a workaround (Check function CastToSheetData). I include the full example below:

program.py

def return_matrix():

    dataTable = []
    for r in range(0,2):
        datalist = []

        for c in range(0,2):
            datalist.append(str(r+c))

        dataTable.append(datalist)
    return dataTable

main.cpp

#include <pybind11/embed.h>
#include <iostream>
#include <list>
#include <string>

typedef std::list<std::list<std::string> > SheetData;

namespace py = pybind11;

SheetData CastToSheetData(PyObject *obj)
{
    SheetData data;
    PyObject *iter = PyObject_GetIter(obj);

    if (!iter)
        return data;
    while (true) {
        std::list<std::string> aux_list;
        PyObject *next = PyIter_Next(iter);
        if (!next) {
            // nothing left in the iterator
            break;
        }
        PyObject *iter2 = PyObject_GetIter(next);
        if (!iter2)
            continue;
        while(true) {
            PyObject *next2 = PyIter_Next(iter2);
            if (!next2) {
                // nothing left in the iterator
                break;
            }
            PyObject* pyStrObj = PyUnicode_AsUTF8String(next2); 
            char* zStr = PyBytes_AsString(pyStrObj); 
            std::string foo(strdup(zStr));
            aux_list.push_back(foo);
            Py_DECREF(pyStrObj);
        }
        data.push_back(aux_list);
    }

    return data;
}


int main()
{
    py::scoped_interpreter guard{};
    py::module calc = py::module::import("program");
    py::object result = calc.attr("return_matrix")();

    SheetData data = CastToSheetData(result.ptr());

    for (auto l : data)
    {
        std::cout << "[ ";
        for(auto s : l)
            std::cout << s <<  " ";
        std::cout << "]" << std::endl;
    }

    return 0;
}

Output:

[ 0 1 ]
[ 1 2 ]

Probably, the best way to go here is to make a custom type_caster using code like the one in the CastToSheetData function inside the load method