I am trying and failing to reproduce and understand a problem I saw where multiprocessing failed when using a python module written in C++. My understanding was that the problem is that multiprocessing needs to pickle the function it is using. So I made my_module.cpp as follows:
#include <pybind11/pybind11.h>
int add(int input_number) {
return input_number + 10;
}
PYBIND11_MODULE(my_module, m) {
m.doc() = "A simple module implemented in C++ to add 10 to a number.";
m.def("add", &add, "Add 10 to a number");
}
After
pip install pybind11
I compiled with:
c++ -O3 -Wall -shared -std=c++11 -fPIC $(python3 -m pybind11 --includes) my_module.cpp -o my_module$(python3-config --extension-suffix)
I can import my_module and it works as expected.
I can test if it can be pickled with:
import my_module
import pickle
# Use the add function
print(my_module.add(5)) # Outputs: 15
# Attempt to pickle the module
try:
pickle.dumps(my_module)
except TypeError as e:
print(f"Pickling error: {e}") # Expected error
which outputs Pickling error: cannot pickle 'module' object as expected.
Now I tested multiprocessing and was surprising that it worked. I was expecting it to give a pickling error.
import my_module
from multiprocessing import Pool
# A wrapper function to call the C++ add function
def parallel_add(number):
return my_module.add(number)
if __name__ == "__main__":
numbers = [1, 2, 3, 4, 5]
try:
# Create a pool of worker processes
with Pool(processes=2) as pool:
results = pool.map(parallel_add, numbers)
print(results) # If successful, prints the results
except Exception as e:
print(f"Multiprocessing error: {e}")
How can I make a Python module in C++ with pybind11 which fails with multiprocessing because of a pickling error?
I am using Linux
I don't think your code tries to pickle a module as-is? If you redefine parallel_add to take a module as an argument, then use a partial to pass my_module into it, you can force Python to do that.
import my_module
from functools import partial
from multiprocessing import Pool
# Same wrapper, but now takes a module as an argument
def parallel_add(module, number):
return module.add(number)
if __name__ == "__main__":
numbers = [1, 2, 3, 4, 5]
with Pool(processes=2) as pool:
results = pool.map(partial(parallel_add, my_module), numbers)
print(results)
This throws the error you were expecting:
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/anerdw/stackoverflow/unpickling.py", line 13, in <module>
results = pool.map(partial(parallel_add, my_module), numbers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/multiprocessing/pool.py", line 367, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/multiprocessing/pool.py", line 774, in get
raise self._value
File "/usr/lib/python3.11/multiprocessing/pool.py", line 540, in _handle_tasks
put(task)
File "/usr/lib/python3.11/multiprocessing/connection.py", line 205, in send
self._send_bytes(_ForkingPickler.dumps(obj))
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: cannot pickle 'module' object
You can also get a pickling-related multiprocessing error much more quickly by cutting out the wrapper and trying to pickle the function directly.
import my_module
from multiprocessing import Pool
if __name__ == "__main__":
numbers = [1, 2, 3, 4, 5]
with Pool(processes=2) as pool:
results = pool.map(my_module.add, numbers)
print(results)
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/anerdw/stackoverflow/unpickling.py", line 8, in <module>
results = pool.map(my_module.add, numbers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/multiprocessing/pool.py", line 367, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/multiprocessing/pool.py", line 774, in get
raise self._value
File "/usr/lib/python3.11/multiprocessing/pool.py", line 540, in _handle_tasks
put(task)
File "/usr/lib/python3.11/multiprocessing/connection.py", line 205, in send
self._send_bytes(_ForkingPickler.dumps(obj))
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: cannot pickle 'PyCapsule' object