How to monitor python's concurrent.futures.ProcessPoolExecutor?

Question

We are using the ProcessPoolExecutor from concurrent.futures in a service that asynchronously receives requests, and does the actual, synchronous processing in the process pool.

Once we ran into the case that the process pool was exhausted, so new requests had to wait until some other processes were finished.

Is there a way to interrogate the process pool for its current usage? That would allow us to monitor their state and do proper capacity planning.

If there isn't, is there any good alternative process pool implementation with an asynchronous interface that supports such monitoring/capacity planning?

Answer 1

The simplest way would be to extend ProcessPoolExecutor with desired behaviour. The example below maintains stdlib interface and does not access implementation details:

from concurrent.futures import ProcessPoolExecutor


class MyProcessPoolExecutor(ProcessPoolExecutor):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._running_workers = 0

    def submit(self, *args, **kwargs):
        future = super().submit(*args, **kwargs)
        self._running_workers += 1
        future.add_done_callback(self._worker_is_done)
        return future

    def _worker_is_done(self, future):
        self._running_workers -= 1

    def get_pool_usage(self):
        return self._running_workers