Search code examples
pythonmultiprocessing

python concurrent.futures ProcessPoolExcutor tasks per child


Is there a way to limit the tasks per child in ProcessPoolExecutor from concurrent.futures module?

(like in multiprocessing.Pool)

https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool

(see maxtasksperchild arg)


Solution

  • The short answer is no. There is no mechanism that keep track of the number of tasks performed by a worker in the concurrent.futures.ProcessPoolExecutor. Thus you cannot do it without modifying the internal code of python.

    A solution could be to implement this feature in loky. This is a library to improve concurrent.futures.ProcessPoolExecutor and fix some of its bugs. It provides a mechanism for worker timeout and it would be easy to implement a task counter that "timeout" when maxtasks is reached. If you need this feature, I suggest putting an issue on the loky repo.