Search code examples
linuxshellsftplftp

lftp mirror possible race condition during file transfer?

I was trying to keep a local folder [lftp_source_folder] and a remote folder [lftp_server_path] in sync using lftp mirror configuration.

Script for running it continously is given below.

while true
do
    lftp -f $BATCHFILE
    sleep 20
done

$BATCHFILE manily consists the below :

# sftp config + 
echo "mirror --Remove-source-files -R /lftp_source_folder /lftp_server_path/" >> $BATCHFILE

But problem is that, I have a script which will keep on moving files to /lftp_source_folder.

Now I am confused that is there a chance of race condition due to this implementation.

For example

  1. my script is moving files to /lftp_source_folder say 50% of file has been moved to /lftp_source_folder and the mv command is interrupted by OS. /lftp_source_folder/new_file.txt [new_file.txt is only 50% of size]
  2. lftp has been invoked by the while loop.
  3. mv command again continues

In step 2, will the lftp upload the file which is 50% completed to lftp server and delete the file ? data will be lost in this case.

If it's race condition, what's the solution ?

Solution

  • If you're moving files within the same filesystem, there's no race condition. mv simply performs a rename() operation, and this is atomic, it's not copying file data.

    But if you're moving between different filesystems, you can indeed get a race condition. This is done as a copy followed by deleting the original, and your script might upload the file to the FTP server when only part of it is copied.

    The solution to this is to move the file first to a temporary folder on the same filesystem as /lftp_source_folder, then move it from there to /lftp_source_folder. So when the mirror script sees it there, it's guaranteed to be complete.