I'm running this simple Python program to merge the output lines from two subprocesses:
import select
from subprocess import PIPE, Popen
import sys
subprocs = [
Popen(cmdline, stdout=PIPE) for cmdline in
[['./repeat', 'abc'], ['./repeat', 'xyz']]
]
while True:
rstreams, _, _ = select.select([p.stdout for p in subprocs], [], [])
for stream in rstreams:
sys.stdout.buffer.write(stream.readline())
… where repeat is a simple script that periodically generates output:
#!/bin/bash
while sleep 1 ; do echo $@ ; done
I'd like the output from the two subprocesses to be merged line-wise, so that the merged output contains lines of abc and xyz in any order, relayed in a timely fashion. (Mixing data, such as abxyzc, is not desired.)
However, what I'm finding is that the Python program above produces no output for a very long time, or until I hit CtrlC.
(I've tried changing stream.readline() to stream.read(1), in case the problem is with readline() hanging, even though I don't want that kind of interleaving, but it didn't help. Happens with Python 3.5.2 on Linux and Python 3.7.6 on macOS.)
Why is select() waiting, when the subprocesses are clearly generating output?
The root reason is that you use sys.stdout.buffer which has a buffer internally, and therefore we could not see the result immediately.
We could explicitly flush the result after write. To put sys.stdout.buffer.flush() after buffer.write(). Or just use print(stream.readline()) rather than sys.stdout.buffer.