Search code examples
linuxpipelibcpiping

What buffers are between commands in a pipeline?


I thought it was 1 buffer, but now it occurs to me that it might be 2.

I mean in a pipeline:

cmd1 | cmd2

cmd2's output might be e.g. line buffered and there is no pipeline there. This should be the buffer managed by libc FILE * stream functions like fwrite(), or is this buffer also used by write(3)? However, I just remembered that pipe(7) talks about the size of the pipe buffers that's apparently controlled in the kernel.

Is stdin buffered, too? Are there 3 buffers, 1 in kernel space and 2 in user space?

I previously thought that read(2) hung when a pipeline buffer was empty, but when stdin is not a pipe but rather a terminal, there i no pipeline buffer, right? If it doesn't have its own buffer, does it check different buffers depending on whether stdin is a pipe or a terminal or a regular file?

EDIT: Changed "how many" to "what" in the question. I hope that's not too big a change. I'm interested in knowing what the buffers are, not just a number.


Solution

  • Is stdin buffered, too? Are there 3 buffers, 1 in kernel space and 2 in user space?

    Yes, in general there could be 3 buffers here: one for cmd1 stdout, one for cmd2 stdin, and one in kernel space.

    I previously thought that read(2) hung when a pipeline buffer was empty, but when stdin is not a pipe but rather a terminal, there i no pipeline buffer, right?

    The read system call blocks when there is no input, but that has nothing to do with stdio buffering.

    The kernel buffer exists regardless of whether the input is from a terminal or not (it would be very inefficient for the kernel to transfer one character at a time).

    By default the stdio library will not buffer terminal input, but the application can change that with explicit calls to e.g. setvbuf.

    A blog post with more details is here.