How to prepend stdout and stderr output with timestamp when redirecting into log files?

Question

In Linux I'm starting a program called $cmd in an init script (SysVInit). I'm already redirecting stdout and stderr of $cmd into two different logfiles called $stdout_log and $stderr_log. Now I also want to add a timestamp in front of every line printed into the logfiles.

I tried to write a function called log_pipe as follows:

log_pipe() {
    while read line; do
        echo [$(date +%Y-%m-%d\ %H:%M:%S)] "$line"
    done
}

then pipe the output of my script into this function and after that redirect them to the logfiles as follows:

$cmd | log_pipe >> "$stdout_log" 2>> "$stderr_log" &

What I get is an empty $stdout.log (stdout) what should be okay, because the $cmd normally doesn't print anything. And a $stderr.log file with only timestamps but without error texts.

Where is my faulty reasoning?

PS: Because the problem exists within an init script I only want to use basic shell commands and no extra packages.

Answer 1

In any POSIX shell, try:

{ cmd | log_pipe >>stdout.log; } 2>&1 | log_pipe >>stderr.log

Also, if you have GNU awk (sometimes called gawk), then log_pipe can be made simpler and faster:

log_pipe() {  awk '{print strftime("[%Y-%m-%d %H:%M:%S]"),$0}'; }

Example

As an example, let's create the command cmd:

cmd() { echo "This is out"; echo "This is err">&2; }

Now, let's run our command and look at the output files:

$ { cmd | log_pipe >>stdout.log; } 2>&1 | log_pipe >>stderr.log
$ cat stdout.log
[2019-07-04 23:42:20] This is out
$ cat stderr.log
[2019-07-04 23:42:20] This is err

The problem

cmd | log_pipe >> "$stdout_log" 2>> "$stderr_log"

The above redirects stdout from cmd to log_pipe. The stdout of log_pipe is redirected to $stdout_log and the stderr of log_pipe is redirected to $stderr_log. The problem is that the stderr of cmd is never redirected. It goes straight to the terminal.

As an example, consider this cmd:

cmd() { echo "This is out"; echo "This is err">&2; }

Now, let's run the command:

$ cmd | log_pipe >>stdout.log 2>>stderr.log
This is err

We can see that This is err is not sent to the file stderr.log. Instead, it appears on the terminal. It is never seen by log_pipe. stderr.log only captures error messages from log_pipe.