Search code examples
pythonmultiprocessingpickleboost-python

Python 3.X Multiprocessing Boost Python Failed


I'm trying to use multiprocessing to map a Boost-wrapped function over multiple cores. This works fine in python 2.7, but is failing in python 3.8. I know the obvious answer: why don't you just use python 2.7 if it works? Well the problem is that the current version of this software is only wrapped for python 3.X. Does anyone know a way around this pickling error for boost-wrapped code in python 3?

multiprocessing.pool.MaybeEncodingError: Error sending result: 
'<multiprocessing.pool.ExceptionWithTraceback object at 0x7ff19e308f40>'. Reason: 
'PicklingError("Can't pickle <class 'Boost.Python.ArgumentError'>: import of module 'Boost.Python' failed")'

Solution

  • This error was not informative. In python 2.7 it was automatically converting a float object to an int object, but in python 3.8 this automatic conversion in the multiprocessing module does not occur. So the parameters being passed to the boost-wrapped c++ code were of the wrong type. Changing them to the correct type fixed this error.