How can I Interrupt Pool.map without losing already computes results using python multiprocessing?

I want to run numerous functions using Pool.map() from the Python multiprocessing module that i can interrupt (for example using ctrl+C ).

I can't find a way that I can do this and retain the data of already completed function evaluation while skipping the pending results. Whether the currently running processes finish or get terminated is not really important.

I have tried the following when interrupting with KeyboardInterrupt, but output would not be defined in this case.

from multiprocessing import Pool
from time import sleep

def f(i):
    sleep(i)
    return i


with Pool() as p:
    try:
        output = p.map(f, range(10))
    except KeyboardInterrupt:
        p.close()

print(output)

Solution

You can't do this with pool.map(). pool.map() will block until the output is ready. As such, it's effectively all or nothing. You also cannot use map_async, because even while that does not block, its output is still all-or-nothing.

You would either have to use the Pool's apply_async method for each task, and manually keep track of output as as it arrives (this will involve polling at some interval), or use Process instances directly, in similar fashion.

In addition, you may have to configure a signal handler in both your main thread and the subprocesses.