Python ctypes: how to define a callback having a buffer pointer and a length in argument?

I have a lib (in source form, if it helps) defining, among other things the following function:

void subscribe(const char* ip_addr,
               uint16_t port,
               uint32_t token,
               void (*cb)(void *buffer, uint32_t length, uint32_t token)
                  );

I defined this in python (v3.10.4, if it matters):

from ctypes import *

so_file = './mylib.so'
lib = CDLL(so_file)

ADDRESS = b"127.0.0.1"
PORT = 21000

data_callback_type = CFUNCTYPE(None, POINTER(c_byte), c_int32, c_int32)

def py_data_callback(buf, bln, tok):
    print(f'Data: [{hexlify(buf, sep=":", bytes_per_sep=4)}] ({bln}|{tok})')

data_callback = data_callback_type(py_data_callback)
ret = lib.subscribe(c_char_p(ADDRESS), c_uint16(PORT), c_int32(42), data_callback)
print(ret)

...

Registration and callback apparently work fine, but I'm receiving the pointer itself in the callback and not the content (I think that's consistent with what I coded) as printout looks like:

Data: [b'0cc2ddd9:c07f0000'] (24|42)

and b'0cc2ddd9:c07f0000' looks suspiciously similar to a pointer (I'm on an amd64 machine).

How do I convince ctypes to return a bytes(24) or, alternatively, given the above pointer how do I access pointed array?

I am new to ctypes and I could have missed something in the docs, but I didn't find the answer there.

Solution

In a callback, if POINTER(c_char) is used for data (esp. if the data contains nulls), then string slicing works to read the data as a Python byte string. It's also recommended to set .argtypes and .restype for functions called by ctypes, so it can convert Python-to-C types correctly (and vice versa):

test.c - working example for reproducible testing:

#include <stdint.h>

#ifdef _WIN32
#   define API __declspec(dllexport)
#else
#   define API
#endif

API void subscribe(const char* ip_addr,
                   uint16_t port,
                   uint32_t token,
                   void (*cb)(void *buffer, uint32_t length, uint32_t token)) {
    cb("\x00\x01\x02\x03\x04\x05\x06\x07",8,token);
}

test.py

from ctypes import *

CALLBACK = CFUNCTYPE(None, POINTER(c_char), c_uint32, c_uint32)

# decorating a Python function makes it usable as a callback
@CALLBACK
def callback(buf, length, tok):
    print(buf[:length])
    print(f'Data: [{buf[:length].hex(sep=":", bytes_per_sep=4)}] ({length}|{tok})')

lib = CDLL('./test')
# correct argument and return types
lib.subscribe.argtypes = c_char_p, c_uint16, c_uint32, CALLBACK
lib.subscribe.restype = None

lib.subscribe(b'127.0.0.1', 21000, 42, callback)

Output:

b'\x00\x01\x02\x03\x04\x05\x06\x07'
Data: [00010203:04050607] (8|42)

EDIT Per comments, below demonstrates how to wrap this all up in a class. Note that if decoration is used on the unbound class method, the callback won't be called correctly due to self being a required parameter. It's also important that if the bound callback is wrapped as shown in __init__, then the callback will be wrapped for the lifetime of the use of the class instance. Do not wrap the instance method as it is passed to the subscribe, e.g.:

self.lib.subscribe(self.ip, self.port, token, CALLBACK(self.callback))

because the wrapped object lifetime will end after the subscribe call. If the callback is referenced later, e.g. by the event call I've added, it would fail. Wrapping the bound instance method in __init__ ensures the wrapping lifetime exists for the lifetime of the instance.

test.c

#include <stdint.h>

#ifdef _WIN32
#   define API __declspec(dllexport)
#else
#   define API
#endif

typedef void (*CALLBACK)(void* buffer, uint32_t length, uint32_t token);

CALLBACK g_cb;
uint32_t g_token;

API void subscribe(const char* ip_addr,
                   uint16_t port,
                   uint32_t token,
                   void (*cb)(void *buffer, uint32_t length, uint32_t token)) {
    g_cb = cb;
    g_token = token;
}

API void event(void* buffer, uint32_t length) {
    if(g_cb)
        g_cb(buffer, length, g_token);
}

test.py

from ctypes import *

CALLBACK = CFUNCTYPE(None, POINTER(c_char), c_uint32, c_uint32)

class Test:

    lib = CDLL('./test')
    lib.subscribe.argtypes = c_char_p, c_uint16, c_uint32, CALLBACK
    lib.subscribe.restype = None

    def __init__(self, ip, port):
        self.ip = ip
        self.port = port
        self.cb = CALLBACK(self.callback)

    def subscribe(self, token):
        self.lib.subscribe(self.ip, self.port, token, self.cb)

    def event(self, data):
        self.lib.event(data, len(data))

    def callback(self, buf, length, tok):
        print(buf[:length])
        print(f'Data: [{buf[:length].hex(sep=":", bytes_per_sep=4)}] ({length}|{tok})')

test = Test(b'127.0.0.1', 21000)
test.subscribe(42)
test.event(b'\x00\x01\x02\x03\x04\x05\x06\x07')

Output:

b'\x00\x01\x02\x03\x04\x05\x06\x07'
Data: [00010203:04050607] (8|42)