KeyboardInterrupt with Python multiprocessing.Pool

Question

I want to write a service that launches multiple workers that work infinitely and then quit when main process is Ctrl+C'd. However, I do not understand how to handle Ctrl+C correctly.

I have a following testing code:

import os
import multiprocessing as mp
    

def g():
    print(os.getpid())
    while True:
        pass
        
        
def main():
    with mp.Pool(1) as pool:
        try:
            s = pool.starmap(g, [[]] * 1)
        except KeyboardInterrupt:
            print('Done')


if __name__ == "__main__":
    print(os.getpid())
    main()

When I try to Ctrl+C it, I expect process(es) running g to just receive SIGTERM and silently terminate, however, I receive something like that instead:

Process ForkPoolWorker-1:
Done
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "test.py", line 8, in g
    pass
KeyboardInterrupt

This obviously means that parent and children processes both raise KeyboardInterrupt from Ctrl+C, further suggested by tests with kill -2. Why does this happen and how to deal with it to achieve what I want?

Answer 1

The signal that triggers KeyboardInterrupt is delivered to the whole pool. The child worker processes treat it the same as the parent, raising KeyboardInterrupt.

The easiest solution here is:

1. Disable the SIGINT handling in each worker on creation
2. Ensure the parent terminates the workers when it catches KeyboardInterrupt

You can do this easily by passing an initializer function that the Pool runs in each worker before the worker begins doing work:

import signal
import multiprocessing as mp

# Use initializer to ignore SIGINT in child processes
with mp.Pool(1, initializer=signal.signal, initargs=(signal.SIGINT, signal.SIG_IGN)) as pool:
    try:
        s = pool.starmap(g, [[]] * 1)
    except KeyboardInterrupt:
        print('Done')

The initializer replaces the default SIGINT handler with one that ignores SIGINT in the children, leaving it up to the parent process to kill them. The with statement in the parent handles this automatically (exiting the with block implicitly calls pool.terminate()), so all you're responsible for is catching the KeyboardInterrupt in the parent and converting from ugly traceback to simple message.