A python script executes an IO-bound function a lot of times (order of magnitude: anything between 5000 and 75000). This is still pretty performant by using
def _iterator(): ... # yields 5000-75000 different names
def _thread_function(name): ...
with concurrent.futures.ThreadPoolExecutor(max_workers=11) as executor:
executor.map(_thread_function, _iterator(), timeout=44)
If a user presses CTRL-C, it just messes up a single thread. I want it to stop launching new threads; and finish the current ongoing threads or kill them instantly, whatever.
How can I do that?
Exception handling in concurrent.futures.Executor.map might answer your question.
In essence, from the documentation of concurrent.futures.Executor.map
If a func call raises an exception, then that exception will be raised when its value is retrieved from the iterator.
As you are never retrieving the values from map(), the exception is never raised in your main thread.
Furthermore, from PEP 255
If an unhandled exception-- including, but not limited to, StopIteration --is raised by, or passes through, a generator function, then the exception is passed on to the caller in the usual way, and subsequent attempts to resume the generator function raise StopIteration. In other words, an unhandled exception terminates a generator's useful life.
Hence if you change your code to (notice the for loop):
def _iterator(): ... # yields 5000-75000 different names
def _thread_function(name): ...
with concurrent.futures.ThreadPoolExecutor(max_workers=11) as executor:
for _ in executor.map(_thread_function, _iterator(), timeout=44):
pass
The InterruptedError will be raised in the main thread, and by passing through the generator (executor.map(_thread_function, _iterator(), timeout=44)) it will terminate it.