Logging in main process when using concurrent futures ProcessPoolExecutor

Question

Any ideas on how can I get forked processes to log to main process ?

My setup is like this:

---

from itertools import repeat
from concurrent.futures import ProcessPoolExecutor

logger = logging.getLogger(__name__)

def worker_process(i, logger):
    logger.info(f"{i}")

with ProcessPoolExecutor(max_workers=4) as executor:
    logger.info("Starting")
    executor.map(worker_process, range(5), repeat(logger))

Answer 1

Your worker processes do log to the main thread. You're not seeing any log output because the default log level for your logger is higher than INFO.

Set the logging to INFO and you'll see output:

from itertools import repeat
from concurrent.futures import ProcessPoolExecutor
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)


def worker_process(i, logger):
    logger.info(f"{i}")


with ProcessPoolExecutor(max_workers=4) as executor:
    executor.map(worker_process, range(5), repeat(logger))

Result:

INFO:__main__:0
INFO:__main__:4
INFO:__main__:3
INFO:__main__:1
INFO:__main__:2