Python multiprocessing.connection.Connection not behaving according to specifications

Question

According to python specifications, recv() method of python Connection, (returned from multiprocessing.Pipe(), throws an EOFError when pipe is empty and the other end of the pipe is closed. (Here is the reference: https://docs.python.org/3.9/library/multiprocessing.html#multiprocessing.connection.Connection.recv)

In the following code, I expect the child process to exit immediately after the pipe is closed on the parent process. In other words, I would not see many still alive! prints and also see a no val (got EOFError) print.

import multiprocessing as mp
import sys
import time
from multiprocessing.connection import Connection


def foo(conn: Connection):
    while True:
        try:
            val = conn.recv()
        except EOFError:
            print("no val (got EOFError)")
            return
        print(f"got val {val}")

if __name__ == "__main__":
    print(f"Version: {sys.version}")
    conn_r, conn_w = mp.Pipe(duplex=False)
    proc=mp.Process(target=foo, args=(conn_r,))
    proc.start()
    conn_r.close()
    for i in range(10):
        time.sleep(0.1)
        conn_w.send(i)
    conn_w.close()
    while True:
        if not proc.is_alive():
            break
        print("still alive!")
        proc.join(timeout=0.1)

But here is what I get: Child process never exits and I never catch the EOFError exception.

$ python3 test_pipe.py
Version: 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110]
got val 0
got val 1
got val 2
got val 3
got val 4
got val 5
got val 6
got val 7
got val 8
still alive!
got val 9
still alive!
still alive!
still alive!
still alive!
still alive!
still alive!
still alive!
still alive!
still alive!
still alive!
still alive!
still alive!
still alive!
still alive!
still alive!
still alive!
still alive!
...

Also observed the same behavior on python 3.10. Am I missing something here? How does it behave on your computer and python version? Does this indicate a bug in python implementation?

Answer 1

Very interesting finding! Just by passing conn_w to foo and closing it there, it started to behave as expected! Perhaps conn_w in the code in the question somehow goes to the memory context of child process and is not closed there.

Here is the version of the code that works fine.

import multiprocessing as mp
import sys
import time
from multiprocessing.connection import Connection


def foo(conn: Connection, close_immediately: Connection):
    close_immediately.close()
    while True:
        try:
            val = conn.recv()
        except EOFError:
            print("no val (got EOFError)")
            return
        print(f"got val {val}")

if __name__ == "__main__":
    print(f"Version: {sys.version}")
    conn_r, conn_w = mp.Pipe(duplex=False)
    proc=mp.Process(target=foo, args=(conn_r, conn_w, ))
    proc.start()
    conn_r.close()
    for i in range(10):
        time.sleep(0.1)
        conn_w.send(i)
    conn_w.close()
    while True:
        if not proc.is_alive():
            break
        print("still alive!")
        proc.join(timeout=0.1)

here is the output now:

$ python3 test_pipe.py 
Version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0]
got val 0
got val 1
got val 2
got val 3
got val 4
got val 5
got val 6
got val 7
got val 8
still alive!
got val 9
no val (got EOFError)

Note: This solves my problem as I needed to receive a reliable EOF when pipe was closed. Although I am still not sure why this behaves this way. if anyone knows, let me know!