pybind11 STL autoconverter breaks std::list pointers

Question

I have a C++ library that manipulates (among other things) a list of wrappers that I have been working on converting to Python using pybind11. The rest of the library operates on a pointer to a list of pointers: std::list<Symbol*>*. The problem is that when attempting to autocast a Python list to this C++ list and then initializing a ParamMap, an object that holds the list on the C++ side, the pointers of the list get all messed up. Inspection in GDB reveals that the "next-object pointers" of all the objects are invalid, and this leads to segfaults when traversing the list.

There is no sign of the objects being deallocated on the C++ side, as neither the destructors for the list container ParamMap nor the list objects Symbol are called. I've deduced that the issue might be Python hyperactively deleting objects C++ is still using, but I've tried object terms like py::return_value_policy::reference and py::keep_alive, and they haven't fixed the problem. What is going wrong here? Unfortunately, changing the list type on the C++ side is not an option, but I would really appreciate some help in making this work on the Python side. Thank you!

Here is some minimal reproduction code:

Symbol.hpp

#include <string>

class Symbol {
private:
        std::string val1;
        int val2;

public:
        Symbol(std::string con1, int con2) : val1(con1), val2(con2) {}
};

ParamMap.hpp

#include <list>
#include "Symbol.hpp"

class ParamMap {
private:
        std::list<Symbol*>* lst;
        int otherData;

public:
        ParamMap(std::list<Symbol*>* symbolList, int dat) : lst(symbolList), otherData(dat)  {}
        std::list<Symbol*>* getSymbols() { return lst; }
        int getOtherData() { return otherData; }
};

Query.cpp

#include <iostream>
#include "ParamMap.hpp"

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
namespace py = pybind11;

void getSymbolListSize(ParamMap* map) {
        std::cout << "Entering query method" << std::endl;
        auto sz = map->getSymbols()->size(); // SEGFAULT OCCURS WHEN GETTING SIZE
        std::cout << "Got size successfully. Size = " << sz << std::endl;
}

PYBIND11_MODULE(list_test, handle) {
        handle.def("getSymbolListSize", &getSymbolListSize);

        py::class_<ParamMap>(handle, "ParamMap")
                .def(py::init<std::list<Symbol*>*, int>(), py::keep_alive<1, 2>())
                .def("getOtherData", &ParamMap::getOtherData)
                .def("getSymbols", &ParamMap::getSymbols);

        py::class_<Symbol>(handle, "Symbol")
                .def(py::init<std::string, int>());
}

test.py

import list_test as p

# Creating a list of some random symbols
symbol_list = []
symbol1 = p.Symbol("Hello", 1)
symbol_list.append(symbol1)
symbol2 = p.Symbol("World", 2)
symbol_list.append(symbol2)

# Creating a parammap and passing it the symbol list
pm = p.ParamMap(symbol_list, 71)

print("Symbol list and ParamMap init'd successfully")

# Here, calling Query.cpp's only method
sz = p.getSymbolListSize(pm)
print(sz)

Answer 1

I don't know a lot about how pybind11 works its magic and therefore I can't help you understanding what is going on. However, I have the feeling that pybind attempts to build the list even though your code only uses a pointer to the list. If I were you I'd consider this a pybind bug and post it as an issue on their github page.

As per your code, doing something like this seems to work (although it's not very clean):

#include <list>
#include "Symbol.hpp"

class ParamMap {
private:
    std::list<Symbol*>* lst;
    int otherData;

public:
    ParamMap(std::list<Symbol*> *symbolList, int dat) : lst(symbolList), otherData(dat)  {
      lst = new std::list<Symbol *>;
      for(auto s : *symbolList) {
        lst->push_back(s);
      }
    }

    ~ParamMap() {
      delete lst;
    }

    std::list<Symbol*>* getSymbols() { return lst; }
    int getOtherData() { return otherData; }
};

I don't know who's supposed to manage the lifetime of the pointed list, so you may want to remove the destructor in case someone else is supposed to deallocate the list.