How to set result of future\add to queue from another thread in Cython?

Question

I have C++ dll which works with multiple threads. So I wrapped this library with Cython and created special receiver callback-function, which must adds some results to asyncio.Queue.

cdef void __cdecl NewMessage(char* message) nogil:

I marked it as nogil, this callback calls from another thread. In this callback I just use:

with gil:
   print("hello")  # instead adding to Queue. print("hello") is more simple operation to demostrate problem

And got deadlock here. How to resolve it?

C++ callback declaration (header):

typedef void (*receiver_t)(char*);
void SetReceiver(receiver_t new_rec);

cpp:

static receiver_t receiver = nullptr;

void SetReceiver(receiver_t new_rec)
{
    printf("setted%i\n", (int)new_rec);
    a = 123;
    if (new_rec != nullptr)
        receiver = new_rec;
}

Cython code:

cdef extern from "TeamSpeak3.h":
    ctypedef void (*receiver_t) (char*) nogil
    cdef void __cdecl SetReceiver(receiver_t new_rec) nogil

cdef void __cdecl NewMessage(char* message) nogil:
    with gil:
        print("hello")

SetReceiver(NewMessage)

Full code: .h http://pastebin.com/ZTCjc6NA

.cpp http://pastebin.com/MeygA8im

.pyx http://pastebin.com/k4X9c54P

.py http://pastebin.com/1YV7tMiF

Answer 1

This is a bit of a guess but you probably have a Cython/C/C++ loop running that's holding the GIL and never releasing it. The callback is then forced for wait forever for it.

In normal Python code the GIL is released every few instructions if another thread is waiting for it. In Cython that doesn't happen automatically. One way to ensure that it does happen every so often is to to your loop:

while True:
   # ... do stuff
   with nogil:
      pass

This ensures the GIL is released once per loop.

Unfortunately it's not obvious to me where you have your main loop. I wonder if it's inside connect in your PyTeamSpeak3 class, and perhaps changing the definition of connect to:

def connect(self):
    with nogil:
       self.thisptr.Connect()

might help?