Search code examples
pythonappendlist-comprehensionpython-multiprocessingmultiprocessing-manager

ValueError: 'Pool not running' when trying to loop the multiprocessing pool


I'm trying to run the same pool three times so that it adds the values in the same list:

import multiprocessing

def foo(name,archive):
    print('hello ', name)
    archive.append(f"hello {name}")

def main():
    max_process = multiprocessing.cpu_count()-1 or 1
    pool = multiprocessing.Pool(max_process)
    manager = multiprocessing.Manager()

    archive = manager.list()

    arguments = ['home','away','draw']
    for _ in range(3):
        [pool.apply_async(foo, args=[name,archive]) for name in arguments]
        pool.close()
        pool.join()
    print(archive)

if __name__ == '__main__':
    main()

The first batch runs perfectly, but when the second batch goes, this error appears:

ValueError: Pool not running

How should I proceed to generate this looping?


Solution

  • As indicated by Nullman in comments, the error was in keeping pool.close() and pool.join() inside the loop, bringing them out, worked perfectly:

    import multiprocessing
    
    def foo(name,archive):
        print('hello ', name)
        archive.append(f"hello {name}")
    
    def main():
        max_process = multiprocessing.cpu_count()-1 or 1
        pool = multiprocessing.Pool(max_process)
        manager = multiprocessing.Manager()
    
        archive = manager.list()
    
        arguments = ['home','away','draw']
        for _ in range(3):
            [pool.apply_async(foo, args=[name,archive]) for name in arguments]
        pool.close()
        pool.join()
        
        print(archive)
    
    if __name__ == '__main__':
        main()