Self-referencing class: concrete python class from C interface

Question

I am trying to design a C interface which could easily be extended in Python (using ctypes). I've used the natural idiom in C:

struct format {
    int (*can_open)(const char *filename);
    struct format * (*open)(const char *filename);
    void (*delete)(struct format *self);
    int (*read)(struct format *self, char *buf, size_t len);
};

It works nicely if I want to extend this interface from C directly:

struct derived /* concrete implementation */
{
    struct format base;
};

But what I would really like to do, is implement this interface from Python using ctypes. Here is what I have so far:

CANOPENFUNC   = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_char_p)
#OPENFUNC     = ctypes.CFUNCTYPE(ctypes.c_void_p, ctypes.c_char_p)
#OPENFUNC     = ctypes.CFUNCTYPE(ctypes.POINTER( python_format ), ctypes.c_char_p)
#DELETEFUNC   = ctypes.CFUNCTYPE(None, ctypes.c_void_p)
#READFUNC     = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_void_p)

def py_canopen_func( string ):
    print "py_canopen_func", string
    return 1

canopen_func   = CANOPENFUNC(py_canopen_func)
#open_func     = OPENFUNC(  py_open_func)
#delete_func   = DELETEFUNC(py_canopen_func)
#read_func     = READFUNC(py_canopen_func)

class python_format(ctypes.Structure):
  _fields_ = (
    ('can_open',  CANOPENFUNC),
    ('open',      OPENFUNC),
    ('delete',    DELETEFUNC),
    ('read',      READFUNC),
  )
  def __init__(self):
    self.can_open = canopen_func
    OPENFUNC    = ctypes.CFUNCTYPE(ctypes.POINTER(python_format), ctypes.c_char_p)
    def py_open_func2( string ):
      print "py_open_func2", string
      return ctypes.byref(self)
    self.open   = OPENFUNC( py_open_func2 )
    #self.delete = delete_func
    #self.read = read_func

Really I am struggling to define the prototype for OPENFUNC here. Technically it should be:

OPENFUNC    = ctypes.CFUNCTYPE(ctypes.POINTER(python_format), ctypes.c_char_p)

However I need to define python_format first, which in turns requires a definition for OPENFUNC.

Bonus point: what would be an actual function implementation ? For instance:

def func( str ): return None

or

def func( str ): i = python_format(); return ctypes.pointer(i)

both gives me:

class python_format(ctypes.Structure):
  pass
OPENFUNC = ctypes.CFUNCTYPE(ctypes.POINTER( python_format ), ctypes.c_char_p)
OPENFUNC( func )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: invalid result type for callback function

Is this related to this other issue ? If so should I change my initial C design, since I will not be able to return a pointer to a python_format instance from a callback ?

Answer 1

In order to have a canonical answer, I'll answer my own question, thanks to @eryksun guidance.

So first thing first, while this is not clear from the documentation, one cannot return complex type from a callback function. Therefore, one cannot map a C function pointer:

struct format {
    struct format * (*open)(const char *filename);
};

to

class python_format:
  pass
OPENFUNC    = ctypes.CFUNCTYPE(ctypes.POINTER(python_format), ctypes.c_char_p)
def py_canopen_func( string ):
    return None
open_func     = OPENFUNC(py_open_func)

The above code will gracefully compiles, but at runtime, one gets:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: invalid result type for callback function

The long answer is that:

The TypeError message you get when trying to use a non-simple type as the result of a callback is less than helpful. A callback's result type has to have a setfunc in its StgDictObject (a ctypes extension of the regular PyDictObject). This requirement restricts you to using a simple type such as c_void_p[...]

Therefore the only solution here, as of today, until issue 5710 (aka 49960) is fixed is the following:

class python_format(ctypes.Structure):
  __self_ref = []
  def __init__(self):
    self.open      = self.get_open_func()
  # technically should be a @classmethod but since we are self-referencing
  # ourself, this is just a normal method:
  def get_open_func(self):
    def py_open_func( string ):
      python_format.__self_ref.append( self )
      return ctypes.addressof(self)
    return OPENFUNC( py_open_func )
OPENFUNC     = ctypes.CFUNCTYPE(ctypes.c_void_p, ctypes.c_char_p)

# delay init required because `read_info` requires a forward declaration:
python_format._fields_ = (
    ('open',      OPENFUNC),
  )