Search code examples
pythonpython-multiprocessing

How to call a function that is inside another function using pool multiprocessing?


I need to run the same function 10 times that for reasons of data linked to a login, it needs to be inside another function:

from multiprocessing import Pool

def main():
    def inside(a):
        print(a)
    Pool.map(inside, 'Ok' * 10)

if __name__ == '__main__':
    main()
from multiprocessing import Pool

def main():
    def inside(a):
        print(a)
    Pool.map(main.inside, 'Ok' * 10)

if __name__ == '__main__':
    main()

In both attempts the result is this:

AttributeError: 'function' object has no attribute 'map'

How can I do this by keeping the function inside the other function?
Is there a way to do this?


Solution

  • AttributeError: 'function' object has no attribute 'map'
    

    We need to instantiate Pool from multiprocessing and call map method of that pool object.

    You have to move inside method to some class because Pool uses pickel to serialize and deserialize methods and if its inside some method then it cannot be imported by pickel.

    Pool needs to pickle (serialize) everything it sends to its worker-processes (IPC). Pickling actually only saves the name of a function and unpickling requires re-importing the function by name. For that to work, the function needs to be defined at the top-level, nested functions won't be importable by the child and already trying to pickle them raises an exception (more).

    Please visit this link of SO.

    from multiprocessing import Pool
    
    class Wrap:
        def inside(self, a):
            print(a)
    
    def main():
        pool = Pool() 
        pool.map(Wrap().inside, 'Ok' * 10)
    
    if __name__ == '__main__':
        main()
    

    If you don't want to wrap inside method inside of a class move the inside method to global scope so it can be pickled

    from multiprocessing import Pool
    
    def inside(a):
        print(a)
    
    def main():
        with Pool() as pool:
            pool.map(inside, 'Ok'*10)
    
    if __name__ == '__main__':
        main()