Search code examples
pythonpython-multiprocessing

Python pool process management


I was running some Python code and realized when I instantiate Pool with only one process, for instance:

from multiprocessing.pool import Pool
from time import sleep

def f(i):
  print(i)
  sleep(10)

with Pool(1) as p:
    p.map(f, [i for i in range(100)])

Actually five processes are currently running. I've also noticed a pattern going on: if I instantiate Pool with 1, 2,3,... processes the number of processes launched by Python are 5,6,7,... I'm curious: Does Pool use three process for management?


Solution

  • with Pool(1), you will get 2 processes, the main process (pid 31070), and one worker process (pid 31071), but 3 extra threads/LWPs in the main process (LWP/thread id 31072/31073/31074):

      PID  PPID   LWP  NLWP CMD
    31070 21240 31070     4 python3 so_48968836_mp.py
    31070 21240 31072     4 python3 so_48968836_mp.py
    31070 21240 31073     4 python3 so_48968836_mp.py
    31070 21240 31074     4 python3 so_48968836_mp.py
    31071 31070 31071     1 python3 so_48968836_mp.py
    

    those three threads are for pool workers maintaining, async task and result handling.