Search code examples
pythonpython-multiprocessingpathos

Pathos pool prints mysterious error

I'm running 

from pathos.multiprocessing import ProcessingPool
pool = ProcessingPool(4)
pool.map(...)

and the following gets printed in run time:

not in depth 2
not in depth 3
not in depth 1
not in depth 2
not in depth 3
not in depth 1
not in depth 2
not in depth 3
not in depth 1

What does this mean? Does it imply any steps I can do to prevent errors increase performance?

Solution

  • That error message is coming from statsmodels, when regression results are being pickled (see issue raised on github).

    ProcessingPool pickles the input when sending to the workers, so that's why this particular message appeared when mapping my function, but not when running it iteratively.