Search code examples
pythonmultiprocessingpool

Several questions about multiprocessing.Pool


I started recently to learn multiprocessing in python. Regarding this I have some questions. The following code shows my example:

import multiprocessing
from time import *

def func(n):
    for i in range(100):
        print(i, "/ 100")
        for j in range(100000):
            a=j*i
            b=j*i/2

if __name__ == '__main__':
    #Test with multiprosessing
    pool = multiprocessing.Pool(processes=4)
    t1 = clock()
    pool.map(func, range(10))
    pool.close()
    t2 = clock()
    print(t2-t1)
    #Test without multiprocessing
    func(range(10))
    t3 = clock()
    print(t3-t2)
  1. Does this code use the four cores of the cpu or did I make a mistake?
  2. Why is the runtime without multiprocessing so much faster? Is there a mistake?
  3. Why does the print command not work while using the multiprocessing?

Solution

  • It does submit four processes at a time to your process pool. Your multiprocessing example is running func ten times, whereas the plain call is running only once. In addition, starting processes has some run time overhead. These probably account for the difference in run times.

    I think a simpler example is instructive. func now sleeps for five seconds, then prints out its input n, along with the time.

    import multiprocessing
    import time
    
    def func(n):
        time.sleep(5)
        print([n, time.time()])
    
    if __name__ == '__main__':
        #Test with multiprosessing
        print("With multiprocessing")
        pool = multiprocessing.Pool(processes=4)
        pool.map(func, range(5))
        pool.close()
    
        #Test without multiprocessing
        print("Without multiprocessing")
        func(1)
    

    pool.map(func, range(5)) runs func(0), func(1), ..., func(4).

    This outputs

    With multiprocessing
    [2, 1480778015.3355303]
    [3, 1480778015.3355303]
    [1, 1480778015.3355303]
    [0, 1480778015.3355303]
    [4, 1480778020.3495753]
    Without multiprocessing
    [1, 1480778025.3653867]
    

    Note that the first four are output at the same time, and not strictly in order. The fifth (n == 4), gets output five seconds later, which makes sense since we had a pool of four processes and it could only get started once the first four were done.