How to handle concurrency if C/C++ Python3 extension hosting a http server and calling python code as request handler?

Question

I try to create a Python extension which will host a HTTP server. When a particular request coming in, the extension calling a Python code from the extension consumer to handle the request. Since HTTP server is by-nature multi-thread, the controller/Python-code will be called in parallel.

According to Python Document, I try to do something like below (removed all error handling code branch to make it simpler)

static string CallingPythonFunc(string input)
{
    PyGILState_STATE gstate;
    gstate = PyGILState_Ensure();

    PyObject *arglist;
    PyObject *result;

    /* Time to call the callback */

    PyGILState_Release(gstate);
    return output;
}

The my_python_func could be very simple like below:

def foo(input):
    return input+"123"

but apparently if there are two requests coming in at the same time, above code will deadlock.

I also found some other sample1, sample2 and StackOverflow answer1, answer2. But seems both sample code create thread by itself, my case the thread is created by the http server. Two StackOverflow answers make feel confused. I thought my code above should wait for GIL available and then run the python call-back code, then get out from the critical section. But seems what I show above is not the case.

Answer 1

Eventually, I got the solution by talking to some guru of Python: I need to call PyEval_InitThreads() and wrap the code which creating non-Python threads with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS.