Search code examples
pythonmultiprocessing

Multiprocessing: pool and map and sys.exit()


I think I need some suggestion here. Below is my code:

from multiprocessing import Pool
import time
import sys

def testing(number):
    count = 0
    while True:
        print('Count: {}'.format(count))
        count += 1

        if count > number:
            print('Exiting...')
            sys.exit()
        else:
            print('Looping Over')
            time.sleep(1)

if __name__ == '__main__':

    with Pool(2) as p:
        p.map(testing, [3, 2])

EXPECTED OUTCOME:

The program (main thread) should exit once all child threads have exited.

ACTUAL RESULT:

$ python3 test_exit.py
Count: 0
Looping Over
Count: 0
Looping Over
Count: 1
Looping Over
Count: 1
Looping Over
Count: 2
Looping Over
Count: 2
Exiting...   <<< Exited 1st thread.
Count: 3
Exiting...   <<< Exited 2nd thread.
....and it stays here as if stuck or something. It never gives control back to Shell.

EXPECTED RESULT:

$ python3 test_exit.py
Count: 0
Looping Over
Count: 0
Looping Over
Count: 1
Looping Over
Count: 1
Looping Over
Count: 2
Looping Over
Count: 2
Exiting...
Count: 3
Exiting...
$   <<< Note: I am expecting to be dropped back to Shell prompt

QUESTION:

Is there something wrong in my approach in term of pool/map usage?


Solution

  • The program (main thread) should exit once all child threads have exited.

    • finish a process with terminating its target function testing() (done through break statement in the crucial loop)
    • exit the main thread/program when the pool of processes is completed.

    from multiprocessing import Pool, current_process
    import time
    import sys
    
    def testing(number):
        count = 0
        while True:
            print('Count: {}'.format(count))
            count += 1
    
            if count > number:
                print('Exiting...', current_process().name)
                break
            else:
                print('Looping Over')
                time.sleep(1)
    
    if __name__ == '__main__':
    
        with Pool(2) as p:
            p.map(testing, [3, 2])
        sys.exit()
    

    The output:

    Count: 0
    Looping Over
    Count: 0
    Looping Over
    Count: 1
    Looping Over
    Count: 1
    Looping Over
    Count: 2
    Looping Over
    Count: 2
    Exiting... ForkPoolWorker-2
    Count: 3
    Exiting... ForkPoolWorker-1
    $