Pass objects between libraries in Python cffi

Question

If I make a new struct with cffi.FFI.new, how do I pass it to a function from a different FFI that has the same struct definition?

I have a basic C struct that I am using in Python via the cffi package that I want to pass to various functions generated and compiled by cffi at runtime. However, I do not know how to get the generated functions to share the same struct definition so that I can pass objects between them. cffi does not like it when building an object with one FFI and passing it to a function from another FFI.

Here is a simplified runnable example of the struct definition and creating an instance in Python:

from cffi import FFI

common_header = """
typedef struct {
  int32_t a;
  double b;
} my_struct;
"""

# FFI for building objects
ffibuilder = FFI()
ffibuilder.cdef(common_header)

# Build an object in Python
my_object = ffibuilder.new('my_struct*')
my_object.a = 3
my_object.b = 2.0

I have an external library that generates the source code of functions that take pointers to instances of this struct. I currently compile them using the API-mode of CFFI. The important thing here is that the functions may be generated after the objects have been constructed, so I cannot simply collect all the functions together ahead of time and compile them as one library.

# Builder for functions generated at runtime
def build_library(header: str, source: str):
    from tempfile import TemporaryDirectory

    ffitemp = FFI()

    ffitemp.cdef(common_header + header)

    ffitemp.set_source('_temp', source)

    with TemporaryDirectory() as temp_dir:
        lib_path = ffitemp.compile(tmpdir=temp_dir)

        lib = ffitemp.dlopen(lib_path)

    return lib.func


# Use function
header = """
int func(my_struct *A);
"""

source = """
typedef struct {
  int32_t a;
  double b;
} my_struct;

int func(my_struct *A) {
    return A -> a;
}
"""

func = build_library(header, source)

When I try to pass instances of my struct to the function, I get an error saying that the struct I am passing in is not the same type as the one accepted by the function.

# Use function
a = func(my_object)
print(a)

TypeError: initializer for ctype 'my_struct *' appears indeed to be 
'my_struct *', the types are different (check that you are not e.g. 
mixing up different ffi instances)

The error is pretty clear about why it is unhappy. It does not like that I have constructed my_object using ffibuilder and passed it to a function defined in a different FFI, which has its own definition of the my_struct type.

How do I get the compilation of the generated functions to share a struct definition with a central FFI?

Answer 1

You can use FFI.include to include the source and definitions of one FFI instance in another. Objects constructed with the included FFI are passable to functions in the FFI in which it was included.

Note that an included definition cannot be duplicated in later FFI. Also, an FFI can only be included if set_source has been called on it. This is true even if all you want is the header; in which case, simply set the source to an empty string.

Here is setting the empty source on the main FFI:

from cffi import FFI

common_header = """
typedef struct {
  int32_t a;
  double b;
} my_struct;
"""

# FFI for building objects
ffibuilder = FFI()
ffibuilder.cdef(common_header)
ffibuilder.set_source('_main', '')  # <-- Set empty source

And here is including the main FFI in the leaf FFI:

# Builder for functions generated at runtime
def build_library(header: str, source: str):
    from tempfile import TemporaryDirectory

    ffitemp = FFI()

    ffitemp.include(ffibuilder)  # <-- include main FFI

    ffitemp.cdef(header)

    ffitemp.set_source('_temp', source)

    with TemporaryDirectory() as temp_dir:
        lib_path = ffitemp.compile(tmpdir=temp_dir)

        lib = ffitemp.dlopen(lib_path)

    return lib.func