Search code examples
shellsedpipefilteringtail

tail -f | sed to file doesn't work

I am having an issue with filtering a log file that is being written and writing the output to another file (if possible using tee, so I can see it working as it goes).

I can get it to output on stdout, but not write to a file, either using tee or >>. I can also get it to write to the file, but only if I drop the -f options from tail, which I need.

So, here is an overview of the commands:

  1. tail -f without writing to file: tail -f test.log | sed 's/a/b/' works
  2. tail writing to file: tail test.log | sed 's/a/b/' | tee -a a.txt works
  3. tail -f writing to file: tail -f test.log | sed 's/a/b/' | tee -a a.txt doesn't output on stdout nor writes to file.

I would like 3. to work.

Solution

  • It's the sed buffering. Use sed -u. man sed:

    -u, --unbuffered

          load  minimal amounts of data from the input files and flush the
          output buffers more often

    And here's a test for it (creates files fooand bar):

    $ for i in {1..3} ; do echo a $i ; sleep 1; done >> foo &
[1] 12218
$ tail -f foo | sed -u 's/a/b/' | tee -a bar
b 1
b 2
b 3

    Be quick or increase the {1..3} to suit your skillz.