Search code examples
pythonmultiprocessingsys

Avoid multiprocessing.Pool to run Globals


I am currently working in Python. I need to work with multiprocessing.Pool and set the current working directory as the file dir an then adding the parent directory to sys.path. The problem is that, when multiprocessing.Pool starts, it runs again the line sys.path.insert(0,'..') which I want to avoid. Is there any way to make multiprocessing.Pool not running the Globals or anything which is outside the if __name__ == "__main__": snippet of code?

import os
import sys
import multiprocessing

os.chdir(sys.path[0])
sys.path.insert(0,'..')
print(os.getcwd())

def Worker(j):
    pass

if __name__ == "__main__":
    with multiprocessing.Pool(1) as p:
        p.map(Worker, range(10))
        p.close()
        p.join()

Solution

  • So the way multiprocessing works is that when it hits the p.map(Worker, range(10)) line, it would start a new child process, and run the whole script again from line 1 (new child process would import the calling module). The lines under the if would not run again in child process. To avoid the sys.path.insert(0,'..') to be repeated in the child process, you need to put that line under the if.

    See this for more details.

    import os
    import sys
    import multiprocessing
    
    
    def Worker(j):
        pass
    
    if __name__ == "__main__":
        os.chdir(sys.path[0])
        sys.path.insert(0,'..')
        print(os.getcwd())
    
        with multiprocessing.Pool(1) as p:
            p.map(Worker, range(10))
            p.close()
            p.join()