I need to wait for multiple processes to finish execution. I am using multiprocess.Process, because I want to adjust oom score of the subprocesses before executing them. After research, I wanted to use multiprocessing.connection.wait and sentinel values of subprocesses to wait on them.
Consider the following code:
import multiprocessing
import multiprocessing.connection
import os
from datetime import datetime
def adj_oom_score_exec(*command: str):
with open("/proc/{}/oom_score_adj".format(os.getpid()), "w") as f:
f.write("500")
os.execv(command[0], command)
exit(198)
pp = multiprocessing.Process(target=adj_oom_score_exec, args=["/usr/bin/sleep", "1"])
pp.start()
print(datetime.now())
multiprocessing.connection.wait([pp.sentinel])
print(datetime.now())
The code outputs two timestamps with difference in milliseconds. multiprocessing.connection.wait() does not wait for the command to finish execution. How to make it wait? Note that pp.join() does work and because subprocess is not a daemon, Python will actually wait for the process to finish.
There is no magic. I was under the impression that multiprocess.Process.sentinel is a pidfd_open returned file descriptor. I suspect that multiprocessing.Process opens a pipe() and sentinel is just a normal pipe. Because python closes the pipe when os.execv, select() on it just returns immediately. Moreover, today I learned that subprocess.Popen.wait() (called by multiprocessing.Process.join()) that is just a busy loop that calls waitpid every 500us.
Instead, use subprocess.Popen with preexec_fn to set the oom_score_adj. In the main thread do threading.Event.wait() and in SIGCHLD signal handler set the event to notify main thread to handle one of the childs.