Search code examples
bashbufferingcoproc

Line buffering in a bash coproc

I'm trying to use a coproc and I'm running into difficulties, most likely with buffering. I have a complex command which accepts line-oriented input from stdin and prints a line to stdout, per line of input. At the command-line this command works fine on a per-line basis, but when I put it in a coproc and read from the ${COPROC[0]} FD, the read blocks.

I can recreate this behavior with paste, but not with cat. I expect paste and cat to pretty much do the same thing when passed no parameters. This is the case when running directly at the command prompt:

$ cat
Hello World!<RETURN>
Hello World!^D
$ paste
Hello World!<RETURN>
Hello World!^D
$

(RETURN and ^D added for illustration)

But when I put these in a , they behave differently - cat is strictly line-buffered, whereas paste appears to be operating with a much larger buffer:

$ coproc cat
$ echo 'Hello world!' >&${COPROC[1]}
$ read -ru ${COPROC[0]} line; echo $line
Hello world!
$ kill $COPROC_PID
[3]+  Terminated              coproc COPROC cat
$ 
$ coproc paste
[3] 42657
$ echo 'Hello world!' >&${COPROC[1]}
$ read -ru ${COPROC[0]} line; echo $line
#### read blocks here until ^C ####

I think the reason for this is that paste adjusts its buffering mode, depending on what it is connected to, whereas cat is always in line-buffering mode.

Is there any way to force paste (or other general command) to be line buffered in the coproc?

After a bit more experimentation I found I can recreate similar behavior without s and instead just piping between cat and paste:

$ cat | cat
Hello World!<RETURN>
Hello World!^D
$ cat | paste
Hello World!<RETURN>
Hello World!^D
$ paste | cat
Hello World!<RETURN>
#### command blocks here until ^C ####

(RETURN and ^D added for illustration)

  • First we pipe cat to cat and get line buffering all the way though
  • Next we pipe cat to paste and also get line buffering all the way though
  • Lastly we pipe paste to cat and don't get line buffering

This appears to indicate that paste will line buffer its stdout when in interactive mode, but otherwise it will use a much bigger buffer.

Solution

  • One way to force line-buffering is to use the stdbuf coreutils tool, if available:

    stdbuf allows one to modify the buffering operations of the three standard I/O streams associated with a program.

    For the coproc case:

    $ coproc stdbuf -oL paste
[3] 42751
$ echo 'Hello world!' >&${COPROC[1]}
$ read -ru ${COPROC[0]} line; echo $line
Hello world!
$ kill $COPROC_PID
[3]+  Terminated              coproc COPROC stdbuf -oL paste
$

    For the simple pipe of paste to cat case:

    $ stdbuf -oL paste | cat
Hello World!<RETURN>
Hello World!^D
$