Scons AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch'

Question

I have an sconstruct script that is instantiating an object.This object internally calls a method than run multiprocessing module. example shown below

This object before calling the function unpickles a file and pass the inputs to multiprocessing module.

def run_scons(self,inpfile,outfile):

        # Unpickle input parameter
        fid=open(inpfile,'rb')
        input_data=pkls.load(fid)
        my_results=[]
        #run solver in loop
        for my_data in input_data:
            work_ers=len(my_data)
            pool = Pool(processes=work_ers)
            a_result=pool.map_async(my_solver, my_data)
            pool.close()
            pool.join()
            my_results.append(a_result.get())
        fid.close()

        fid_out=open(outfile,'wb+')
        pkls.dump(rot_full_results,fid_out)

I get the following error when executing the same function via scons.

pool = Pool(processes=work_ers)
  File "C:\Python27\lib\multiprocessing\__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "C:\Python27\lib\multiprocessing\pool.py", line 138, in __init__
    self._setup_queues()
  File "C:\Python27\lib\multiprocessing\pool.py", line 232, in _setup_queues
    from .queues import SimpleQueue
  File "C:\Python27\lib\multiprocessing\queues.py", line 48, in <module>
    from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition
  File "C:\Python27\lib\multiprocessing\synchronize.py", line 48, in <module>
    from multiprocessing.forking import assert_spawning, Popen
  File "C:\Python27\lib\multiprocessing\forking.py", line 60, in <module>
    class ForkingPickler(Pickler):
  File "C:\Python27\lib\multiprocessing\forking.py", line 61, in ForkingPickler
    dispatch = Pickler.dispatch.copy()
AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch'
scons: building terminated because of errors.

After reading about this error, i found out that SCONS has a hack around where it renames pickle module as cPickle, whereas multiprocessing module is looking for cPickle and everything goes down. Is there any way around this?

Answer 1

I am using multiprocessing to process some 'job' files in parallel. I am doing something like this in my SConstruct file:

if GetOption("run_jobs"):

    my_jobs = # some code that produces a list of job objects

    ecode = build_support.mp_run_jobs(my_jobs)

    Exit(ecode)

But then I ran into the exception above:

AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch'

On Linux using the 'SCONS_HORRIBLE_REGRESSION_TEST_HACK' works:

$ SCONS_HORRIBLE_REGRESSION_TEST_HACK=1 scons --run-jobs

But I don't want to have to export that symbol on Linux and Windows and Mac etc.

I did find a workaround, delete the pickle modules and re-import them like this:

if GetOption("run_jobs"):

    # Workaround SCons.compat module renaming

    import imp

    del sys.modules['pickle']
    del sys.modules['cPickle']

    sys.modules['pickle'] = imp.load_module('pickle', *imp.find_module('pickle'))
    sys.modules['cPickle'] = imp.load_module('cPickle', *imp.find_module('cPickle'))

    import pickle
    import cPickle

    print "(pickle == cPickle) = ", (pickle == cPickle)

    my_jobs = # some code that produces a list of job objects

    ecode = build_support.mp_run_jobs(my_jobs)

    Exit(ecode)

Now my multiprocessing of jobs works as expected!