I have a log file in a custom binary format sitting on a remote machine. I want to sync it with a file on my local machine, so when new bytes are appended to the remote binary file, the file on my local machine will be updated as well.
I learned about how to use tail over ssh here: https://serverfault.com/questions/608723/tail-a-file-from-ssh-and-mirror-to-a-local-file
Then I learned about how to use tail on binary files here: Binary "tail" a file
I tried combining them into
ssh -t remotebox "tail -c +1 -f /path/to/file.bin" > ./mirror.bin
But then I realized that mirror.bin is corrupted. Looking at the hex dump, I see that all 0d0a bytes are truncated into 0as (\r\n got replaced by just \n). However, if I run tail locally (tail -c +1 -f file1.bin > file2.bin), this truncation does not happen.
I also tried to use tee instead of redirection; the problem persists.
Are there shell tricks I can do with tail to prevent this from happening, or are there other programs that suites my needs?
Thanks.
P.S. Both remote and local machines are Linux, running bash.
The following reproduces the problem:
$ echo -e '123\r\n234\r\n' > /tmp/1
$ scp /tmp/1 server:/tmp/1
$ ssh -t server cat /tmp/1 > /tmp/2
Shared connection to closed.
$ diff /tmp/1 /tmp/2
1,3c1,3
< 123
< 234
<
---
> 123
> 234
>
It's possible that ssh -t changes \r\n to \n because it might do stty +igncr on the line. Mine ssh -t does the opposite (I think stty +inlcr) and I have the character doubled:
$ ssh -t server cat /tmp/1 | hexdump -C
Shared connection to closed.
00000000 31 32 33 0d 0d 0a 32 33 34 0d 0d 0a 0d 0a |123...234.....|
0000000e
$ ssh server cat /tmp/1 | hexdump -C
00000000 31 32 33 0d 0a 32 33 34 0d 0a 0a |123..234...|
0000000b
Anyway. remove the -t option if you do not have the intention to use terminal features.
And your command line suggest that you are copying a file. I have no idea why do you use tail for that - tail it's for outputting last part of file, not for printing the whole file... To print the whole file use cat. To copy a file, use scp.