Use extern "Python" style cffi callbacks with embedded PyPy

Question

Can PyPy be embedded use the new extern "Python" style cffi callbacks? PyPy's documentation only shows the old style cffi callbacks but the cffi documentation suggests against using them. The PyPy documentation makes no reference to the new style callbacks and I can't get the new style callbacks to work.

Embedding PyPy

Extern “Python” (new-style callbacks)

# file "interface.py"

import cffi

# add new extern "Python" declaration
ffi = cffi.FFI() ffi.cdef('''
struct API {
    double (*add_numbers)(double x, double y);

    extern "Python" void add_numbers2(double, double);
}; ''')

# Better define callbacks at module scope, it's important to
# keep this object alive.
@ffi.callback("double (double, double)")
def add_numbers(x, y):
    return x + y

# new function
@ffi.def_extern()
def add_numbers2(x, y):
    return x + y


def fill_api(ptr):
    global api
    api = ffi.cast("struct API*", ptr)
    api.add_numbers = add_numbers

Errors when running compiled C (source for C is same as PyPy docs):

debug: OperationError:
debug: operator-type: CDefError
debug: operator-value: cannot parse "extern "Python" void add_numbers2(double, double);"
:6:5: before: extern
Error calling pypy_execute_source_ptr!

Answer 1

"extern Python" is not really meant to be used right now in embedding situations like the one you're pointing to. To support this case nicely, a little more efforts are needed from the developers of cffi (including me :-). In other words, a future cffi release should give an alternative way to do embedding, simpler than both CPython's and PyPy's custom solutions (respectively, "using the CPython C API" and "following https://pypy.readthedocs.org/en/latest/embedding.html"). It should also give a single common solution. However, right now it's not done.

You could apply the existing (cffi 1.4) "extern Python" solution on top of the PyPy documentation's example, but it requires some refactorings---notably, the example uses the "in-line ABI mode", whereas "extern Python" only works with the "out-of-line API mode". If we regard https://pypy.readthedocs.org/en/latest/embedding.html as describing a purely ABI mode approach, then using ffi.callback() is still the documented and only way to do it in CFFI.

Update: CFFI 1.5 has full support for embedding (http://cffi.readthedocs.org/en/latest/embedding.html) in the "extern Python" style. It is available on CPython right now. PyPy requires trunk versions (or PyPy 4.1, which should be out in March or April 2016).