Search code examples
cronrsync

rsync: remote sync file‘s create-time(became to 1970) is different from local file's creation time

I used RSYNC to upload a folder from local host to remote host. It was set to a scheduled task by CRONTAB. And an error occurred during uploading: a file's (in folder) transmission failed, and its creation time became to 1970/01/01(obviously incorrect). Also, the folder's(on remote host) creation time is different from local folder's(on local host) creation time. The former is a few minutes later than the latter. Here is my code(which is set to be a scheduled task by crontab):

time=$(date "+%Y-%m-%d")
echo $time
count_tmp=0
while [ $count_tmp -le 4 ]
do

/usr/bin/expect <<-EOF
spawn rsync -avzP -e "ssh -p xxx" --bwlimit=5000 $time xxx@xxxx:/xxx/xxx
set timeout 300
expect {
 "*yes/no" { send "yes\r"; exp_continue }
 "*password:" { send "changeme\n" }

}
send "exit\r"
expect eof
EOF

if [ $? -ne 0 ]
then
 let count_tmp++
 echo "failed, try again"
 sleep 1h
else
   echo "success"
   break
fi
done

And following is : 2020-08-01 is the folder, and xxx-2020-08-01.txt are files in it. records_history_information-2020-08-01.txt is uploading when error occurred and process interrupted.

-remote host folder's(2020-08-01) info

drwxr-xr-x 2 hgd hgd 4096 Aug  1 20:25 2020-08-01

-remote host file's(records_history_information-2020-08-01.txt) info

-rw-rw-rw- 1 hgd hgd  58182912 Jan  1  1970 records_history_information-2020-08-01.txt

-local host folder's(2020-08-01) info

drwxr-xr-x 2 root root       4096 Aug  1 20:16 2020-08-01

-local host file's(records_history_information-2020-08-01.txt) info

-rw-rw-rw- 1 mysql mysql 727396510 Aug  1 20:16 records_history_information-2020-08-01.txt

local crontab log(show records_history_information-2020-08-01.txt upload interrupted, next is 2020-08-02)

2020-08-01/records_history_information-2020-08-01.txt
2020-08-0292   8%   13.28MB/s    0:00:48

I have two questions:

-The records_history_information-2020-08-01.txt's incorrect creation time(which is 1970) is the result of its fail during uploading?

-The host never shuts down (both) and I have set 5-times re-upload. Why does it close uploading? Could it be caused by the timeout of Rsync? Or other reason?

Solution

  • January 1st, 1970 is the start of the Unix "epoch" (i.e. time zero), so seeing that in the case of a failure sounds plausible. Rerunning rsync with -t or -a should fix it.

    Your expect script has a 5 minute timeout. If that's insufficient time to complete the transfer, or if there's an extra user prompt you haven't anticipated, then yes it will interrupt the upload.

    I suggest you use a higher timeout value for the "expect eof".

    A few other notes:

    • The "send exit" seems unnecessary.
    • The "yes/no" thing looks like a bad idea; if ssh doesn't know the host then something fishy is going on, and you probably shouldn't be connecting blindly.
    • Embedding a password in a script is not ideal; you should set up an ssh key for automatic log in (see ssh-keygen and ssh-copy-id). This will eliminate the need for expect completely (you can use the timeout tool instead).