Search code examples
pythonc++ctbb

Python C, tbb, calling a function from multiple threads


As a learning process of Python C API I am trying to call a Python function inside of functor passed to tbb parallel_for. Operation of calling the function crashes instance of Python process. I am not doing anything not thread safe. I get the item from a list and I then call a Python function with the item passed as an argument to the function. In the end I set the item back to the list. Any hints what have I done wrong ?


Solution

  • It is most likely that you forgot to grab Global Interpreter Lock (GIL) when you call Python function back from C++. For example, TBB module for Python implements this using :

    class PyCaller : public swig::SwigPtr_PyObject {
    public:
      using swig::SwigPtr_PyObject::SwigPtr_PyObject; // gets constructors
    
      void operator()() const {
        SWIG_PYTHON_THREAD_BEGIN_BLOCK;
        PyObject* r = PyObject_CallFunctionObjArgs((PyObject*)*this, NULL);
        if(r) Py_DECREF(r);
        SWIG_PYTHON_THREAD_END_BLOCK;
      }
    };
    
    // Usage:
    tbb::task_group tg;
    void enqueue( PyObject *c ) {
        tg.run( PyCaller(c) );
    }
    

    And you can see how SWIG implements it - here.

    Other options to consider include using Numba's @cfunc(nopython=True) decorator and Cython's nogil attribute which both make function faster and enable Python's function to run in parallel as they remove GIL from the compiled function.