Pybind11: Why doesn't asyn call from Python execute the callbacks properly in C++?

Question

I have a Python method which is implemented like this:

def start(self):
    try:
        self.is_running = True
        self._main_loop()

    except Exception as ex:
        path='exceptions-servicecore.log'
        track = traceback.format_exc()
        exception_time = datetime.now().strftime("%d/%m/%Y %H:%M:%S")
        with open(path, 'a') as f:
            f.writelines(f'\n{exception_time} : exception occured {ex.args} \n{track}')

def start_async(self):
    st = threading.Thread(target=self.start) 
    st.start()

The _main_loop() here then runs bunch of commands and executes the callbacks which are sent from C++/C# etc. Using Pybind11 I simply call start_async like this :

this->startFuncAsync = this->obj.attr("start_async");
...
CORE_API void Core::Start(bool async)
{
    try
    {
        if (async)
        {
            this->startFuncAsync();
        }
        else
        {
            this->startFunc();
        }
    }
    catch (std::exception ex)
    {
        std::cout << ex.what() << std::endl;
    }
}

But it seems, it just doesn't run properly.

It works fine in Python, but when it comes to C++ or C# which call the C++ function above it doesn't show any output! The Python sections which call the callbacks do work as I log their execution to a text file. however there is no activity on the client sides. I don't see any outputs to be produced in C++ or C#. The callbacks are working and outputting the needed information on the console (C++ or C#). The non-async function (start) however, works just fine.

So my question is, is this an expected behavior? Should this not work regardless of how the function is invoked in Python or outside of it?

Answer 1

This issue happened to be related to Pybind11 it seems. That is, only one thread of execution can work and all operations must executed under one same thread.
If the method is started as a separate thread in python like what we have here (start), calling other methods in c++ will use a different thread and thus you can interact with it, it will become like a zombie thread, as you will not be able to interact with it or even stop it.
The logical way would be to execute this in C++ side where you have control over the thread.