Pybind11 - Bound class method returns new class instance, rather than editing in-place

Question

I am unable to return the input of a class method (input: specific instances of a seperate class) to Python. The binding compiles and I can use the resulting module in Python. The class method should however return the same instances as it admits (after some processing).

The Obstacle class is used as the input. The ObstacleProcess class has a method (Python: __call__ / C++: operator_py) which processes the input (instances of Obstacle). The following Python code shows that different instances of Obstacle is returned:

import example

obstacle_1 = example.Obstacle()
obstacle_2 = example.Obstacle()

obstacles = [obstacle_1, obstacle_2]
print(obstacles)

params = example.Params()
obstacle_process = example.ObstacleProcess(params)
obstacles = obstacle_process(obstacles)
print(obstacles)

The first print returns: [<example.Obstacle object at 0x7fb65271e1b0>, <example.Obstacle at 0x7fb652735070>], whilst the second print retuns: [<example.Obstacle at 0x7fb652734670>, <example.Obstacle object at 0x7fb652735230>].

This is not the desired output as obstacle_1 initially lives at 0x7fb65271e1b0, and after the operator()/__call__ call it lives at 0x7fb652734670. I want obstacle_1 to keep its initial address of 0x7fb65271e1b0, even after the other class (ObstacleProcess) has processed obstacle_1.

The following code shows the source code is bound with pybind11:

// pybind11 binding
py::class_<Obstacle, std::shared_ptr<Obstacle>>(m, "Obstacle")
    .def(py::init<>());

py::class_<ObstacleProcess>(m, "ObstacleProcess")
    .def(py::init<
        const Params&>()
    )
    .def("__call__", &ObstacleProcess::operator_py<Params>, py::return_value_policy::reference);

The next block shows how operator_py is implemented in the source code:

template <Params>
std::vector<Obstacle>& operator_py(
    std::vector<Obstacle>& obstacles,
    const Params &parameters
)
{

    ...

    return obstacles
}

I have tried with and without std::shared_ptr<Obstacle>. The current implementation gives the same result as not using shared_ptr at all, thus there is something wrong with how I have implemented shared_ptr. I have tried to use PYBIND11_MAKE_OPAQUE(std::shared_ptr<std::vector<Obstacle>>);, but my implementation of this did not change the result.

I have not tried to use the pybind11 — smart_holder branch, maybe this branch has to be used for this case?

Answer 1

Thanks to comments from @DanMašek and @n.m.willseey'allonReddit it was unveiled that making these changes to operator_py() fixes the issue in the question:

template <class Params>
std::vector<std::shared_ptr<Obstacle>>& operator_py(
    std::vector<std::shared_ptr<Obstacle>> &obstacles,
    const Params &params
    )
    {
    for (auto& obstacle : obstacles)
        {
             obstacle->someObstacleClassMethod()
             someObstacleProcessMethod(obstacles, params)
        }   
    return obstacles;
    }

With this implemented the desired output was obtained:

import example

obstacle_1 = example.Obstacle()
obstacle_2 = example.Obstacle()

obstacles = [obstacle_1, obstacle_2]
print(obstacles)

params = example.Params()
obstacle_process = example.ObstacleProcess(params)
obstacles = obstacle_process(obstacles)
print(obstacles)

# output:
[<example.Obstacle object at 0x7f709bdc2430>, <example.Obstacle object at 0x7f709b12a830>]
[<example.Obstacle object at 0x7f709bdc2430>, <example.Obstacle object at 0x7f709b12a830>]