Flushing a pipe (os.pipe) before closing

Question

I need to launch a subprocess and enable two threads for reading its stdout and stderr respectively.

The following code is just considering stdout:

def reader(rfd):
    while True:
        try:
            data = os.read(rfd, bufsize)
        except OSError:
            break
        else:
            chomp(data)

rout, wout = os.pipe()
tout = threading.Thread(target=reader, args=(rout,))
tout.start()

subprocess.check_call(command, bufsize=bufsize, stdout=wout, stderr=werr)

os.close(wout)
os.close(rout)
tout.join()

The code works, except I noticed that not all data is processed, as if the os.close(wout) function kills the reader before all data is read. On the other hand, if I don't close wout my process will be hanging forever on tout.join().

I can tell this is a buffering problem because if I put a very bad time.sleep(0.1) just after subprocess.check_call(...) everything magically works.

The good way would be flushing instead of waiting, but any call to os.fsync() over a pipe is giving OSError: [Errno 22] Invalid argument.

Any hint about how to flush a pipe created with os.pipe?

Answer 1

I would recommend using Popen rather than os.pipe for interprocess communication.

eg.

writer_process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
reader_thread = threading.Thread(target=reader, args=(writer_process.stdout,))
reader_thread.start()
reader_thread.join()

However, if you really want to use os.pipe then you will have an easier time treating them like file objects. Python's in-built file context manager will make sure the files and flushed and closed appropriately.

eg.

def reader(fd):
    with os.fdopen(fd, bufsize=bufsize) as f:
        while True:
            data = f.read(bufsize)
            if not data:
                break
            chomp(data)

and

with os.fdopen(wout, "w", bufsize=bufsize) as f:
    subprocess.check_call(cmd, stdout=f)