Pooling a member function in python 2.7

Question

I am getting a weird error with this code. I am trying to pool instances of a worker function which is a member the class invoking the pool. While I had my doubts if this will work or not, I am not sure of the exact reason as to why? The error thrown when I run this is a "PicklingError". Can someone explain why?

import multiprocessing
import time


class Pooler(multiprocessing.Process):
  def __init__(self):
    multiprocessing.Process.__init__(self)

  def run(self):
    pool = multiprocessing.Pool(10)
    print "starting pool"
    pool.map(self.worker, xrange(10), chunksize=10)

  def worker(self, arg):
    print "worker - arg - {}".format(arg)


if __name__ == '__main__':
  jobs = []
  for i in range(5):
    proc = Pooler()
    jobs.append(proc)
    proc.start()

  for j in jobs:
    j.join()

  print "...ending"

UPDATE

I changed the code to look as follows:

import multiprocessing
import time


class Pooler(multiprocessing.Process):
  def __init__(self):
    multiprocessing.Process.__init__(self)

  def run(self):
    pool = multiprocessing.Pool(1)
    print "starting pool"
    obj = Worker()
    pool.map(obj.run, range(10), chunksize=1)

class Worker(object):
  def __init__(self):
    pass

  def run(self, nums):
    print "worker - arg - {}".format(nums)


if __name__ == '__main__':
  jobs = []
  for i in range(1):
    proc = Pooler()
    jobs.append(proc)
    proc.start()

  for j in jobs:
    j.join()

  print "...ending"

but I am still getting the following error:

starting pool
Process Pooler-1:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "pool_test.py", line 13, in run
    pool.map(obj.run, range(10), chunksize=1)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 567, in get
    raise self._value
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed
...ending

Answer 1

The answer is simple. multiprocessing uses pickle to serialize objects and pass those objects between the different processes -- and as the error states, pickle can't serialize an instancemethod. You need to use a better serializer, like dill if you want to serialize an instancemethod (see https://stackoverflow.com/a/21345273/2379433).

So what can you do about multiprocessing? Fortunately, there is a fork of multiprocessing called multiprocess that uses dill, and if you use it, your object will serialize and your code will work. It's a one-line change that comes with the bonus of being able to run from the interpreter as well as serialize almost all the objects in python. (The link I posted above is for pathos and dill, but pathos is built on top of multiprocess, so it's still very relevant.)

>>> import multiprocess as multiprocessing
>>> import time
>>> class Pooler(multiprocessing.Process):
...   def __init__(self):
...     multiprocessing.Process.__init__(self)
...   def run(self):
...     pool = multiprocessing.Pool(1)
...     print "starting pool"
...     obj = Worker()
...     pool.map(obj.run, range(10), chunksize=1)
... 
>>> class Worker(object):
...   def __init__(self):
...     pass
...   def run(self, nums):
...     print "worker - arg - {}".format(nums)
... 
>>> if __name__ == '__main__':
...   jobs = []
...   for i in range(1):
...     proc = Pooler()
...     jobs.append(proc)
...     proc.start()
...   for j in jobs:
...     j.join()
...   print "...ending"
... 
starting pool
worker - arg - 0
worker - arg - 1
worker - arg - 2
worker - arg - 3
worker - arg - 4
worker - arg - 5
worker - arg - 6
worker - arg - 7
worker - arg - 8
worker - arg - 9
...ending
>>>