I've spent several hours on research and no solution seems to work anymore. My function will take an estimated 30-50 minutes per process. If I spot flaws during the first few minutes, I don't want to wait for the processes to finish completely or have to close the terminal without killing the processes cleanly.
For simplicity, each process is assigned a function with an infinite loop. With Ctrl+c the complete main program should now be interrupted without getting stuck. The reason for the infinite loop is that every previous solution has worked with lists that are actually looped through. The keyboard interrupt is recognized, but only afterwards, when all processes have finished. I don't know if this has anything to do with the Python version, but that's not the requirement I would like.
I would like a simple and clean solution to this problem. That would probably help many others as well.
Example:
from multiprocessing import Pool
def f(x):
while True:
pass
if __name__ == '__main__':
with Pool(2) as p:
p.map(f, [1, 2])
In another forum this problem was discussed and solved in a simple and clean way. Here is the link for everyone else:
In multiprocessing module there is an AsyncResult object. This includes various functions such as get(), wait() and ready(). While get() and wait() block, you can use ready() to wait for the result and still catch KeyboardInterrupts.
Unfortunately, only map_async() and apply_async() return an AsyncResult object.
from multiprocessing import Pool
def f(x):
while True:
pass
if __name__ == "__main__":
with Pool(2) as p:
result = p.map_async(f, [1, 2])
while not result.ready():
time.sleep(1)
This way you can also use functions that block:
Playing around with signals it seems that from inside child processes, exceptions other than KeyboardInterrupt or SystemExit propagate to the main process. When ctrl+c is pressed, all processes (children and main process) run their SIGINT handler which by default just raises KeyboardInterrupt. To get control out of the children, the child interrupt handler must raise some other exception. The exception is not seen by the way because the main process has its hands full with its own KeyboardInterrupt.
Basically, the SIGINT handler can also return None.
from multiprocessing import Pool
from signal import signal, SIGINT
def initializer():
signal(SIGINT, lambda: None)
def f(x):
while True:
pass
if __name__ == '__main__':
with Pool(n, initializer) as p:
p.map(f, [1, 2])