Search code examples
pythonsubprocesspopen

python subprocess: "write error: Broken pipe"

I have a problem piping a simple subprocess.Popen.

Code:

import subprocess
cmd = 'cat file | sort -g -k3 | head -20 | cut -f2,3' % (pattern,file)
p = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE)
for line in p.stdout:
    print(line.decode().strip())

Output for file ~1000 lines in length:

...
sort: write failed: standard output: Broken pipe
sort: write error

Output for file >241 lines in length:

...
sort: fflush failed: standard output: Broken pipe
sort: write error

Output for file <241 lines in length is fine.

I have been reading the docs and googling like mad but there is something fundamental about the subprocess module that I'm missing ... maybe to do with buffers. I've tried p.stdout.flush() and playing with the buffer size and p.wait(). I've tried to reproduce this with commands like 'sleep 20; cat moderatefile' but this seems to run without error.

Solution

  • From the recipes on subprocess docs:

    # To replace shell pipeline like output=`dmesg | grep hda`
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]