scp stuck without any error until I move my terminal to another monitor

We have a simple script which will run at 4 AM everyday to copy the latest backup of our production database and restore it on our staging database:

/usr/bin/trickle -s -d 30720 -u 30720 -t 10 -l 100 /usr/bin/scp -r root@$MARIADB_REMOTE_HOST:$MARIADB_REMOTE_HOME $MARIADB_DATA_DIR || RESPONSE=$? && true

The source ($MARIADB_REMOTE_HOST) is a bare-metal server and the destination is a VM ran by KVM (OpenNebula). Both of them running Ubuntu 18.04 .

Last week we changed our network switch which was connecting our bare-metals and after that the script will stuck at some point in the middle of the scp. It doesn't follow any pattern and doesn't exit; It just stuck and when I move my terminal to another monitor, It start again (continue) and obviously It's not a solution !!!

I tried to run it with "-v" but it didn't help (The last line was printed by the script due to the SIGTERM that I've sent):

Sending file modes: C0640 561 delivery_category_equivalency.frm.qp
Sink: C0640 561 delivery_category_equivalency.frm.qp
Sending file modes: C0640 3817 order_config_prioritization.ibd.qp
Sink: C0640 3817 order_config_prioritization.ibd.qp
Sending file modes: C0640 482 item_service.frm.qp
Sink: C0640 482 item_service.frm.qp
Sending file modes: C0640 429 feedback_record.frm.qp
Sink: C0640 429 feedback_record.frm.qp
Sending file modes: C0640 381409 order_configs.ibd.qp
Sink: C0640 381409 order_configs.ibd.qp
Sending file modes: C0640 35810042703 terminals.ibd.qp
Sink: C0640 35810042703 terminals.ibd.qp
debug1: SSH2_MSG_KEXINIT received
debug1: SSH2_MSG_KEXINIT sent
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: rekeying in progress
debug1: rekeying in progress
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:ngRaDyCEeh6BBJFLMC7xq2Cl5h/ICky2/+Hbn/UpUzY
debug1: ssh_set_newkeys: rekeying out, input 49398838524 bytes 134202402 blocks, output 8768056 bytes 19126 blocks
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: ssh_set_newkeys: rekeying in, input 49398838536 bytes 134202403 blocks, output 8768056 bytes 0 blocks
debug1: rekey in after 134217728 blocks
debug1: SSH2_MSG_KEXINIT received
debug1: SSH2_MSG_KEXINIT sent
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: rekeying in progress
debug1: rekeying in progress
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:ngRaDyCEeh6BBJFLMC7xq2Cl5h/ICky2/+Hbn/UpUzY
debug1: ssh_set_newkeys: rekeying out, input 50472727740 bytes 134203417 blocks, output 8960060 bytes 19238 blocks
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: ssh_set_newkeys: rekeying in, input 50472727752 bytes 134203418 blocks, output 8960060 bytes 0 blocks
debug1: rekey in after 134217728 blocks
debug1: SSH2_MSG_KEXINIT received
debug1: SSH2_MSG_KEXINIT sent
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: rekeying in progress
debug1: rekeying in progress
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:ngRaDyCEeh6BBJFLMC7xq2Cl5h/ICky2/+Hbn/UpUzY
debug1: ssh_set_newkeys: rekeying out, input 51546616956 bytes 134203417 blocks, output 9149904 bytes 19022 blocks
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: ssh_set_newkeys: rekeying in, input 51546616968 bytes 134203418 blocks, output 9149904 bytes 0 blocks
debug1: rekey in after 134217728 blocks
debug1: SSH2_MSG_KEXINIT received
debug1: SSH2_MSG_KEXINIT sent
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: rekeying in progress
debug1: rekeying in progress
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:ngRaDyCEeh6BBJFLMC7xq2Cl5h/ICky2/+Hbn/UpUzY
debug1: ssh_set_newkeys: rekeying out, input 52620506172 bytes 134203417 blocks, output 9340548 bytes 19102 blocks
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: ssh_set_newkeys: rekeying in, input 52620506184 bytes 134203418 blocks, output 9340548 bytes 0 blocks
debug1: rekey in after 134217728 blocks
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 2 clearing O_NONBLOCK
Killed by signal 15.
        Removing old MariaDB's files failed at "20210722-0055" with 1

It goes without saying that when it stuck, I don't see any traffic between these 2 servers when I inspect it via iftop or tcpdump.

I will appreciate it if you share your idea about how to address this issue.

Solution

Test your SCP with these options

scp -o ServerAliveInterval=15 -o ServerAliveCountMax=3 <yourServerIP>:~

In this example, every 15 seconds looking back for acks and if 3 acks in 45 seconds (15*3) not coming back close the session.