How to reduce buffering / cause auto-flush when viewing AWS Cloudwatch logs with --follow option

Question

The AWS cli command logs is v useful for keeping up with what our instances are doing

I use the following to view our application logs while:

• filtering out the HTTP requests
• filtering out the HealthChecks

aws --follow --region=ap-south-1 logs tail "/ECS-CLUSTER/" --since 5h
       | grep -v "GET /" | grep -v "POST /" | grep -v HealthCh

This works well - except that the last one to two KB of logs are not displayed. They are apparently not flushed out. Is there a knob to turn to disable the buffering or to ensure the content is transmitted after some maximum wait say 1000ms?

Answer 1

Just in case anyone finds this old question, I had the same problem, which is caused by aws logs tail detecting that its output is not a tty and buffering it. If logs are infrequent it can take a long time for the buffer to fill and output to be sent.

It's possible hack around this using expect's unbuffer (but not stdbuf -o0 -e0), but if that's too heavy-weight I wrote a tiny tool to solve this: github.com/cyanogilvie/forcetty.