Pool and Manager in multiprocessing module

Question

from multiprocessing import Pool, Manager


def test(num):
    queue.put(num)


queue = Manager().Queue()
pool = Pool(5)

for i in range(30):
    pool.apply_async(test, (i, ))

pool.close()
pool.join()

print(queue.qsize())

The output of the code above is 30. However, if line 8 is exchanged with line 9 (see the code below), the output will be 0. So is there anybody who knows why? Thank you!

 from multiprocessing import Pool, Manager


 def test(num):
   queue.put(num)


 pool = Pool(5)
 queue = Manager().Queue()

for i in range(30):
    pool.apply_async(test, (i, ))

pool.close()
pool.join()

print(queue.qsize())

Answer 1

It happens due to fact that you are sharing queue via globals. So if you do pool init with queue already created it shares the same Queue, otherwise not. Add print globals() to test method and you will see difference.

Nevertheless, it's better to pass queue to pool as method param to be sure that it's "the same" object.

def test(num, q):
    q.put(num)

def main():
    pool = Pool(5)
    q = Manager().Queue()

    for i in range(30):
        pool.apply_async(test, (i, q))

    pool.close()
    pool.join()

    print(q.qsize())
if __name__ == '__main__':
    main()

Output: 30