Pass a closure from Cython to C++

Question

I have a C++ function that accepts a callback, like this:

void func(std::function<void(A, B)> callback) { ... }

I want to call this function from Cython by giving it a closure, i.e. something I would have done with a lambda if I was calling it from C++. If this was a C function, it would have some extra void* arguments:

typedef void(*callback_t)(int, int, void*);

void func(callback_t callback, void *user_data) {
    callback(1, 2, user_data);
}

and then I would just pass PyObject* as user_data (there is a more detailed example here).

Is there way to do this more in C++ way, without having to resort to explicit user_data?

Answer 1

What I believe you're aiming to do is pass a callable Python object to something accepting a std::function. You need to do create a bit of C++ code to make it happen, but it's reasonably straightforward.

Starting by defining "accepts_std_function.hpp" as simply as possible to provide an illustrative example:

#include <functional>
#include <string>

inline void call_some_std_func(std::function<void(int,const std::string&)> callback) {
    callback(5,std::string("hello"));
}

The trick is then to create a wrapper class that holds a PyObject* and defines operator(). Defining operator() allows it to be converted to a std::function. Most of the class is just refcounting. "py_obj_wrapper.hpp":

#include <Python.h>
#include <string>
#include "call_obj.h" // cython helper file

class PyObjWrapper {
public:
    // constructors and destructors mostly do reference counting
    PyObjWrapper(PyObject* o): held(o) {
        Py_XINCREF(o);
    }
    
    PyObjWrapper(const PyObjWrapper& rhs): PyObjWrapper(rhs.held) { // C++11 onwards only
    }
    
    PyObjWrapper(PyObjWrapper&& rhs): held(rhs.held) {
        rhs.held = 0;
    }
    
    // need no-arg constructor to stack allocate in Cython
    PyObjWrapper(): PyObjWrapper(nullptr) {
    }
    
    ~PyObjWrapper() {
        Py_XDECREF(held);
    }
    
    PyObjWrapper& operator=(const PyObjWrapper& rhs) {
        PyObjWrapper tmp = rhs;
        return (*this = std::move(tmp));
    }
    
    PyObjWrapper& operator=(PyObjWrapper&& rhs) {
        std::swap(held, rhs.held);
        return *this;
    }
    
    void operator()(int a, const std::string& b) {
        if (held) { // nullptr check 
            call_obj(held,a,b); // note, no way of checking for errors until you return to Python
        }
    }
    
private:
    PyObject* held;
};

This file uses a very short Cython file to do the conversions from C++ types to Python types. "call_obj.pyx":

from libcpp.string cimport string

cdef public void call_obj(obj, int a, const string& b):
    obj(a,b)

You then just need to create the Cython code wraps these types. Compile this module and call test_func to run this. ("simple_version.pyx":)

cdef extern from "py_obj_wrapper.hpp":
    cdef cppclass PyObjWrapper:
        PyObjWrapper()
        PyObjWrapper(object) # define a constructor that takes a Python object
             # note - doesn't match c++ signature - that's fine!
             
cdef extern from "accepts_std_func.hpp":
    void call_some_std_func(PyObjWrapper) except +
            # here I lie about the signature
            # because C++ does an automatic conversion to function pointer
            # for classes that define operator(), but Cython doesn't know that
            

def example(a,b):
    print(a,b)
    
def test_call():
    cdef PyObjWrapper f = PyObjWrapper(example)
    
    call_some_std_func(f)

---

The above version works but is somewhat limited in that if you want to do this with a different std::function specialization you need to rewrite some of it (and the conversion from C++ to Python types doesn't naturally lend itself to a template implementation). One easy way round this is to use the Boost Python library object class, which has a templated operator(). This comes at the cost of introducing an extra library dependency.

First defining the header "boost_wrapper.hpp" to simplify the conversion from PyObject* to boost::python::object

#include <boost/python/object.hpp>

inline boost::python::object get_as_bpo(PyObject* o) {
    return boost::python::object(boost::python::handle<>(boost::python::borrowed(o)));
}

You then just need to Cython code to wrap this class ("boost_version.pyx"). Again, call test_func

cdef extern from "boost_wrapper.hpp":
    cdef cppclass bpo "boost::python::object":
        # manually set name (it'll conflict with "object" otherwise
        bpo()
        
    bpo get_as_bpo(object)
    
    
cdef extern from "accepts_std_func.hpp":
    void call_some_std_func(bpo) except + # again, lie about signature

def example(a,b):
    print(a,b)
    
def test_call():
    cdef bpo f = get_as_bpo(example)
    
    call_some_std_func(f)

---

A "setup.py"

from distutils.core import setup, Extension
from Cython.Build import cythonize

extensions = [
    Extension(
           "simple_version",                       # the extension name
           sources=["simple_version.pyx", "call_obj.pyx" ],
           language="c++",                        # generate and compile C++ code
      ),
    Extension(
           "boost_version",                       # the extension name
           sources=["boost_version.pyx"],
           libraries=['boost_python'],
           language="c++",                        # generate and compile C++ code
      )
    ]

setup(ext_modules = cythonize(extensions))

---

(A final option is to use ctypes to generate a C function pointer from a Python callable. See Using function pointers to methods of classes without the gil (bottom half of answer) and http://osdir.com/ml/python-cython-devel/2009-10/msg00202.html. I'm not going to go into detail about this here.)