When I spin up a new server I use a cloud-init script to automate the process. I follow progress of this process with:
$ tail -f /var/log/cloud-init-output.log
When it comes to fetching a backup file using scp and piping it through tar -zx I tried the following in the cloud-init script:
$ scp myuser@123.45.6.789:path/to/file.tar.gz /dev/stdout | tar -zx
Whilst this command works, the handy progress indications that scp outputs do not appear in the tail -f output... i.e. I do not see progress like:
file.tar.gz 57% 52MB 25.2MB/s 00:02
I have also tried bash process substitution like so:
$ scp myuser@123.45.6.789:path/to/file.tar.gz >(tar -zx)
And still the progress indications do not appear in the tail -f output.
How do I preserve the progress indications in the tail -f output? Particularly to see progress when fetching the larger backup files, it would be really handy to preserve these indications.
Note that when I run both of the above in a bash script directly (with set -x at the top), progress does show for the 'bash process substitution' variant but not the 'piping' variant... but when tailing the cloud-init logs, progress shows for neither variant.
It looks like cloud-init just sends both stdout and stderr from all cloud-init stages to /var/log/cloud-init-output.log (see here).
So we can re-create the cloud-init process (for the purposes of this question) with the following call:
$ scp myuser@123.45.6.789:path/to/file.tar.gz >(tar -zx) >output.log 2>&1
and we then follow this log file separately with:
$ tail -f output.log
Strange thing is... no progress status ever appears in this output.log file. And yet progress status is definitely being sent... the only change in the following is to direct to /dev/stdout rather than output.log:
$ scp myuser@123.45.6.789:path/to/file.tar.gz >(tar -zx) >/dev/stdout 2>&1
file.tar.gz 57% 52MB 25.2MB/s 00:02
So why do we see progress status on /dev/stdout but not in output.log when we direct stdout to output.log?
Like some other tools, scp checks if its output is going to a TTY (by use of isatty), disabling the progress meter output if it is not (you can find similar cases e.g., ls --color=auto emits color codes only when output goes to a terminal).
You can trick it into thinking it is outputting to a TTY by running it under script (as shown in this answer), or any other tool (e.g expect) that runs a program with its output connected to a PTY.