I have a function where I call n processes using Pool.starmap(). I want to terminate all the processes after any k out of n are completed. How can I implement this?
This is done more easily with either Pool.imap_unordered or with concurrent.futures.as_completed on a list of submitted tasks. In either case, the solution is the same; iterate the resulting iterator k times (e.g. with itertools.slice), then terminate the Pool (in the Executor case, call shutdown(cancel_futures=True), or make sure you cancel all the outstanding tasks manually). For example, instead of:
with Pool() as pool:
results = pool.starmap(func, makes_tuples())
which collects all the results as a single operation and removes your ability to stop processing midstream, you do:
from itertools import islice
# Defined at global scope as simple wrapper to allow non-starmap functions to
# be used with iterators of argument tuples
def starcall_func(args):
return func(*args)
...
with Pool() as pool:
results = list(islice(pool.imap_unordered(starcall_func, makes_tuples()), k))
# When with exits, pool is terminated automatically, and only k results were collected