Is there a way to limit the tasks per child in ProcessPoolExecutor from concurrent.futures module?
(like in multiprocessing.Pool)
https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool
(see maxtasksperchild arg)
The short answer is no. There is no mechanism that keep track of the number of tasks performed by a worker in the concurrent.futures.ProcessPoolExecutor
. Thus you cannot do it without modifying the internal code of python.
A solution could be to implement this feature in loky. This is a library to improve concurrent.futures.ProcessPoolExecutor
and fix some of its bugs. It provides a mechanism for worker timeout and it would be easy to implement a task counter that "timeout" when maxtasks
is reached. If you need this feature, I suggest putting an issue on the loky repo.