Search code examples
pythonexceptionmultiprocessingpicklepython-multiprocessing

Weird unpickling error when using multiprocessing

I get the following error when using multiprocessing:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 282, in _handle_results
    task = get()
UnpicklingError: NEWOBJ class argument has NULL tp_new

I have absolutely no idea what this means, although it sounds like something's wrong at the C level. Can anyone shed some light on this?

UPDATE: Ok, so I figured out how to fix this. But I'm still a bit perplexed. I'm returning an instance of this class:

class SpecData(object):
    def __init__(self, **kwargs):
        self.__dict__.update(**kwargs)
    def to_dict(self):
        return self.__dict__

If I return an instance of this object, I get the error. However, if I call to_dict and return a dictionary, it works. What am I doing wrong?

Solution

  • Try using the pickle module rather than the cPickle module -- pickle is written in pure Python, and often it gives more useful error messages than cPickle. (Though sometimes I've had to resort to making a local copy of pickle.py, and adding in a few debug printf statements near the location of the error to figure out the problem.)

    Once you track down the problem, you can switch back to cpickle.

    (I'm not that familiar with the multiprocessing module, so I'm not sure whether you're doing the pickling or it is. If it is, then the easiest way to get it to use pickle rather than cpickle may be to do some monkey-patching before you import the multiprocessing/threading module: import sys, pickle; sys.modules['cPickle']=pickle)