Search code examples
bashgreppipeio-buffering

bash, chaining grep commands, processing as the data come


When I use a single grep command, it processes and output the data live as it comes.

Here is my simple test file test.sh:

echo a
sleep 1
echo b
sleep 1
echo ab
sleep 1
echo ba
sleep 1
echo baba

I do the following:

sh test.sh | grep a
a
ab
ba
ab
ba

all good so fa. 'a' appears immediately, then 'ab', etc.

But when I pipe multiple grep commands like that

sh ./test.sh | grep a | grep b
ab
ba
baba

I only get the output at the end, not as it comes! the terminal stays empty till the entire file is processed then outputs everything in one go.

Why is that?

How can i chain/cascade multiple greps without losing that 'process and output as it comes' property?

This is for greping and processing live huge logs with a lot of data where I only have a chance to save to disk the filtered version and not the huge raw ouput that would fill up the disk quite quickly.


Solution

  • There's an option called line-buffered:

    Other Options
           --line-buffered
                  Use line buffering on output.  This can cause a performance penalty.
    

    So:

    sh ./test.sh | grep --line-buffered a | grep b