I am integrating a legacy C++ library with Python using boost-python. The legacy library has some global initialization and then the classes in it use application wide data. I need to ensure that the shutdown function of the legacy library is called after all wrapped objects are destroyed and thought this might be achieved by registering a shutdown function using atexit. However, I found that the wrapped objects are being cleaned up after atexit calls the shutdown function, causing multiple segfaults within the legacy library!
I can achieve the desired behavior by calling del on the wrapped objects before exiting, but was hoping to leave deletion to Python. I have checked out the red warning box in the documentation of object.__del__, and am wondering if my ideal world is unreachable.
Any suggestions for ensuring a shutdown method is called after all objects are cleaned up when wrapping legacy code in a python module?
Some platform details in case they are important:
Minimal code:
#include <iostream>
#include <boost/python.hpp>
using namespace std;
namespace legacy
{
void initialize() { cout << "legacy::initialize" << endl; }
void shutdown() { cout << "legacy::shutdown" << endl; }
class Test
{
public:
Test();
virtual ~Test();
};
Test::Test() { }
Test::~Test() { cout << "legacy::Test::~Test" << endl; }
}
BOOST_PYTHON_MODULE(legacy)
{
using namespace boost::python;
legacy::initialize();
class_<legacy::Test>("Test");
def("_finalize", &legacy::shutdown);
object atexit = object(handle<>(PyImport_ImportModule("atexit")));
object finalize = scope().attr("_finalize");
atexit.attr("register")(finalize);
}
Once compiled, this can be run using python with the following input and outputs being displayed:
>>> import legacy
legacy::initialize
>>> test = legacy.Test()
>>> ^Z
legacy::shutdown
legacy::Test::~Test
In short, create a guard type that will initialize and shutdown the legacy library in its constructor and destructor, then manage the guard via a smart pointer in each exposed object.
There are some subtle details that can make getting the destruction process correct difficult:
Py_Finalize() is random.To accomplish this, the Boost.Python objects need to coordinate when to initialize and shutdown the legacy API. These objects also need to have ownership over the legacy object that uses the legacy API. Using the single responsibility principle, one can divide the responsibilities into a few classes.
One can use the resource acquisition is initialization (RAII) idiom to initialize and shutdown the legacy AP. For example, with the following legacy_api_guard, when a legacy_api_guard object is constructed, it will initialize the legacy API. When the legacy_api_guard object is destructed, it will shutdown the legacy API.
/// @brief Guard that will initialize or shutdown the legacy API.
struct legacy_api_guard
{
legacy_api_guard() { legacy::initialize(); }
~legacy_api_guard() { legacy::shutdown(); }
};
As multiple objects will need to share management over when to initialize and shutdown the legacy API, one can use a smart pointer, such as std::shared_ptr, to be responsible for managing the guard. The following example lazily initializes and shutdown the legacy API:
/// @brief Global shared guard for the legacy API.
std::weak_ptr<legacy_api_guard> legacy_api_guard_;
/// @brief Get (or create) guard for legacy API.
std::shared_ptr<legacy_api_guard> get_api_guard()
{
auto shared = legacy_api_guard_.lock();
if (!shared)
{
shared = std::make_shared<legacy_api_guard>();
legacy_api_guard_ = shared;
}
return shared;
}
Finally, the actual type that will be embedded into the Boost.Python object needs to obtain a handle to the legacy API guard before creating an instance of the legacy object. Additionally, upon destruction, the legacy API guard should be released after the legacy object has been destroyed. One non-intrusive way to accomplish this is to use provide a custom HeldType when exposing the legacy types to Boost.Python. When exposing the type, the default Boost.Python generated initializers need to be suppressed, as a custom factory function will be used instead to provide control over object creation:
/// @brief legacy_object_holder is a smart pointer that will hold
/// legacy types and help guarantee the legacy API is initialized
/// while these objects are alive. This smart pointer will remain
/// transparent to the legacy library and the user-facing Python.
template <typename T>
class legacy_object_holder
{
public:
typedef T element_type;
template <typename... Args>
legacy_object_holder(Args&&... args)
: legacy_guard_(::get_api_guard()),
ptr_(std::make_shared<T>(std::forward<Args>(args)...))
{}
legacy_object_holder(legacy_object_holder& rhs) = default;
element_type* get() const { return ptr_.get(); }
private:
// Order of declaration is critical here. The guard should be
// allocated first, then the element. This allows for the
// element to be destroyed first, followed by the guard.
std::shared_ptr<legacy_api_guard> legacy_guard_;
std::shared_ptr<element_type> ptr_;
};
/// @brief Helper function used to extract the pointed to object from
/// an object_holder. Boost.Python will use this through ADL.
template <typename T>
T* get_pointer(const legacy_object_holder<T>& holder)
{
return holder.get();
}
/// Auxiliary function to make exposing legacy objects easier.
template <typename T, typename ...Args>
legacy_object_holder<T>* make_legacy_object(Args&&... args)
{
return new legacy_object_holder<T>(std::forward<Args>(args)...);
}
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<
legacy::Test, legacy_object_holder<legacy::Test>,
boost::noncopyable>("Test", python::no_init)
.def("__init__", python::make_constructor(
&make_legacy_object<legacy::Test>))
;
}
Here is a complete example demonstrating using a custom HeldType to non-intrusively lazily guard a resource with shared management:
#include <iostream> // std::cout, std::endl
#include <memory> // std::shared_ptr, std::weak_ptr
#include <boost/python.hpp>
/// @brief legacy namespace that cannot be changed.
namespace legacy {
void initialize() { std::cout << "legacy::initialize()" << std::endl; }
void shutdown() { std::cout << "legacy::shutdown()" << std::endl; }
class Test
{
public:
Test() { std::cout << "legacy::Test::Test()" << std::endl; }
virtual ~Test() { std::cout << "legacy::Test::~Test()" << std::endl; }
};
void use_test(Test&) {}
} // namespace legacy
namespace {
/// @brief Guard that will initialize or shutdown the legacy API.
struct legacy_api_guard
{
legacy_api_guard() { legacy::initialize(); }
~legacy_api_guard() { legacy::shutdown(); }
};
/// @brief Global shared guard for the legacy API.
std::weak_ptr<legacy_api_guard> legacy_api_guard_;
/// @brief Get (or create) guard for legacy API.
std::shared_ptr<legacy_api_guard> get_api_guard()
{
auto shared = legacy_api_guard_.lock();
if (!shared)
{
shared = std::make_shared<legacy_api_guard>();
legacy_api_guard_ = shared;
}
return shared;
}
} // namespace
/// @brief legacy_object_holder is a smart pointer that will hold
/// legacy types and help guarantee the legacy API is initialized
/// while these objects are alive. This smart pointer will remain
/// transparent to the legacy library and the user-facing Python.
template <typename T>
class legacy_object_holder
{
public:
typedef T element_type;
template <typename... Args>
legacy_object_holder(Args&&... args)
: legacy_guard_(::get_api_guard()),
ptr_(std::make_shared<T>(std::forward<Args>(args)...))
{}
legacy_object_holder(legacy_object_holder& rhs) = default;
element_type* get() const { return ptr_.get(); }
private:
// Order of declaration is critical here. The guard should be
// allocated first, then the element. This allows for the
// element to be destroyed first, followed by the guard.
std::shared_ptr<legacy_api_guard> legacy_guard_;
std::shared_ptr<element_type> ptr_;
};
/// @brief Helper function used to extract the pointed to object from
/// an object_holder. Boost.Python will use this through ADL.
template <typename T>
T* get_pointer(const legacy_object_holder<T>& holder)
{
return holder.get();
}
/// Auxiliary function to make exposing legacy objects easier.
template <typename T, typename ...Args>
legacy_object_holder<T>* make_legacy_object(Args&&... args)
{
return new legacy_object_holder<T>(std::forward<Args>(args)...);
}
// Wrap the legacy::use_test function, passing the managed object.
void legacy_use_test_wrap(legacy_object_holder<legacy::Test>& holder)
{
return legacy::use_test(*holder.get());
}
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<
legacy::Test, legacy_object_holder<legacy::Test>,
boost::noncopyable>("Test", python::no_init)
.def("__init__", python::make_constructor(
&make_legacy_object<legacy::Test>))
;
python::def("use_test", &legacy_use_test_wrap);
}
Interactive usage:
>>> import example
>>> test1 = example.Test()
legacy::initialize()
legacy::Test::Test()
>>> test2 = example.Test()
legacy::Test::Test()
>>> test1 = None
legacy::Test::~Test()
>>> example.use_test(test2)
>>> exit()
legacy::Test::~Test()
legacy::shutdown()
Note that the basic overall approach is also applicable to a non-lazy solution, where the legacy API gets initialized upon importing the module. One would need to use a shared_ptr instead of a weak_ptr, and register a cleanup function with atexit.register():
/// @brief Global shared guard for the legacy API.
std::shared_ptr<legacy_api_guard> legacy_api_guard_;
/// @brief Get (or create) guard for legacy API.
std::shared_ptr<legacy_api_guard> get_api_guard()
{
if (!legacy_api_guard_)
{
legacy_api_guard_ = std::make_shared<legacy_api_guard>();
}
return legacy_api_guard_;
}
void release_guard()
{
legacy_api_guard_.reset();
}
...
BOOST_PYTHON_MODULE(example)
{
// Boost.Python may throw an exception, so try/catch around
// it to initialize and shutdown legacy API on failure.
namespace python = boost::python;
try
{
::get_api_guard(); // Initialize.
...
// Register a cleanup function to run at exit.
python::import("atexit").attr("register")(
python::make_function(&::release_guard)
);
}
// If an exception is thrown, perform cleanup and re-throw.
catch (const python::error_already_set&)
{
::release_guard();
throw;
}
}
See here for a demonstration.