Search code examples
pythoncmultithreadinggilpython-embedding

Embedding python in multithreaded C application

I'm embedding the python interpreter in a multithreaded C application and I'm a little confused as to what APIs I should use to ensure thread safety.

From what I gathered, when embedding python it is up to the embedder to take care of the GIL lock before calling any other Python C API call. This is done with these functions:

gstate = PyGILState_Ensure();
// do some python api calls, run python scripts
PyGILState_Release(gstate);

But this alone doesn't seem to be enough. I still got random crashes since it doesn't seem to provide mutual exclusion for the Python APIs.

After reading some more docs I also added: 

PyEval_InitThreads();

right after the call to Py_IsInitialized() but that's where the confusing part comes. The docs state that this function:

Initialize and acquire the global interpreter lock

This suggests that when this function returns, the GIL is supposed to be locked and should be unlocked somehow. but in practice this doesn't seem to be required. With this line in place my multithreaded worked perfectly and mutual exclusion was maintained by the PyGILState_Ensure/Release functions.
When I tried adding PyEval_ReleaseLock() after PyEval_ReleaseLock() the app dead-locked pretty quickly in a subsequent call to PyImport_ExecCodeModule().

So what am I missing here?

Solution

  • Eventually I figured it out.
    After 

    PyEval_InitThreads();

    You need to call 

    PyEval_SaveThread();

    While properly release the GIL for the main thread.