As a learning process of Python C API I am trying to call a Python function inside of functor passed to tbb parallel_for. Operation of calling the function crashes instance of Python process. I am not doing anything not thread safe. I get the item from a list and I then call a Python function with the item passed as an argument to the function. In the end I set the item back to the list. Any hints what have I done wrong ?
It is most likely that you forgot to grab Global Interpreter Lock (GIL) when you call Python function back from C++. For example, TBB module for Python implements this using swig:
class PyCaller : public swig::SwigPtr_PyObject {
public:
using swig::SwigPtr_PyObject::SwigPtr_PyObject; // gets constructors
void operator()() const {
SWIG_PYTHON_THREAD_BEGIN_BLOCK;
PyObject* r = PyObject_CallFunctionObjArgs((PyObject*)*this, NULL);
if(r) Py_DECREF(r);
SWIG_PYTHON_THREAD_END_BLOCK;
}
};
// Usage:
tbb::task_group tg;
void enqueue( PyObject *c ) {
tg.run( PyCaller(c) );
}
And you can see how SWIG implements it - here.
Other options to consider include using Numba's @cfunc(nopython=True)
decorator and Cython's nogil
attribute which both make function faster and enable Python's function to run in parallel as they remove GIL from the compiled function.