Async python function as a C callback

Question

I'm in a situation where I need to use a C library that calls a python function for IO purposes so I don't have to create a C equivalent for the IO. The python code base makes extensive use of asyncio and all the IO goes through queues. And the C code is loaded as a dll using ctypes The problem is you can't await from a C callback. Is there anyway to use an async python function as a C callback?

Below is essentially how it will work. Is my only option to do something like the non_async_callback or is there someway to await from C using ctypes.

python

import asyncio
import time
import ctypes

DLL_PATH="./bin/test.dll"

dll = ctypes.CDLL(DLL_PATH)
incoming_queue=asyncio.Queue()
outgoing_queue=asyncio.Queue()

c_func =  ctypes.CFUNCTYPE(ctypes.c_uint8, ctypes.c_uint8)

async def python_callback(test_num):
    await outgoing_queue.put(test_num)
    test_num = await incoming_queue.get()
    return test_num

def non_async_callback(test_num):
    while True:
        try:
            outgoing_queue.put_nowait(test_num)
            break
        except asyncio.QueueFull:
            time.sleep(0.1)
            continue
   
    while True:
        try:
            test_num = incoming_queue.get_nowait(test_num)
            break 
        except asyncio.QueueFull:
            time.sleep(0.1)
            continue

    return test_num

# Called at some point during initialization
def setup_callback():
    dll.setup_callback(c_func(python_callback))

C

uint8_t (CALLBACK*)(uint8_t);
CALLBACK py_callback

void setup_callbacks(void* callback)
{
    py_callback = callback;
}

// Called from somewhere else in the C code.
uint8_t python_callback(uint8_t value)
{
    uint8_t result = py_callback(value);
    // Do something with result
}

Answer 1

From a bit of searching, I believe

async def python_callback(test_num):
    await outgoing_queue.put(test_num)
    test_num = await incoming_queue.get()
    return test_num

can be wrapped with something like

loop = None  # No loop has been created initially

def non_async_callback(test_num):
    global loop

    if not loop:  # Only create a new loop on first callback
        loop = asyncio.new_event_loop()

    test_num = loop.run_until_complete(python_callback(test_num))
    return test_num

If non_async_callback might be called concurrently from multiple threads, then just use:

def non_async_callback(test_num):
    test_num = asyncio.run(python_callback(test_num))
    return test_num

Update

Based on the OP's comment it appears that we initially have a Python script calling a C function and that C function ultimately results in a callback being invoked. Thus the callback is occurring on the same thread that the original Python script (and thus its event loop) is executing. Consequently, the problem becomes calling a Python coroutine from a regular Python function without blocking the current event loop. This requires creating a separate thread running a separate event loop:

import asyncio
import threading

# Create a new thread running a new event_loop:
_new_event_loop = asyncio.new_event_loop()

# Run the new event loop on a new daemon thread:
threading.Thread(target=_new_event_loop.run_forever, name="Async Runner", daemon=True).start()

async def python_callback(test_num):
    print('starting')
    await asyncio.sleep(1) # Emulate doing something
    print('ending')
    return test_num * test_num

def non_async_callback(test_num):
    # Invoke the callback on a different thread, since no
    # further awaits can be done on this thread until this
    # funtion returns:
    return asyncio.run_coroutine_threadsafe(python_callback(test_num), _new_event_loop).result()

async def main():
    ...
    # Ultimately our callback gets invoked on the same thread:
    test_num = non_async_callback(4)
    print(test_num)

asyncio.run(main())

Prints:

starting
ending
16