Search code examples
sortingrsync

`rsync` files-from in a numerical order


I have thousand of files named as 1.tld, 2.tld, 3.tld, ... 3550.tld, ...
Because I am using a mobile modem which has a SIM card with a limited amount of Mbytes of data transfer I manually CTRL-C a rsync command when I see I'm near that limit.
I want to know which files were synced, let's say from 1.tld to 7510.tld.
But rsync doesn't sync in a numerical order, it starts with 1.tld, 10.tld, 100.tld, 1000.tld, 10000.tld ... so I have synced the file 10000.tld but probably not the 7000.tld.
So I ls -tr > list-of-files.txt and rsync -avz —files-from=list-of-files.txt source dest.
Despite cat list-of-files.txt displays the list in numerical order rsync does not respect that.

Summary: I want rsync sync in a numerical order: 1.tld, 2.tld, 3.tld, ... as the first file in a verbose lists


Solution

  • From the man page:

    Rsync always sorts the specified filenames into its internal transfer list. This handles the merging together of the contents of identically named directories, makes it easy to remove duplicate filenames, and may confuse someone when the files are transferred in a different order than what was given on the command-line.

    If you need a particular file to be transferred prior to another, either separate the files into different rsync calls, or consider using --delay-updates (which doesn't affect the sorted transfer order, but does make the final file-updating phase happen much more rapidly).

    You might:

    • Rename the source files so they're zero-padded like 00001.tld instead of 1.tld. This should cause them to be sorted correctly by rsync.
    • Issue 5 rsync calls, first for all 1-digit files like rsync from/?.tld to, then for all 2-digit files rsync from/??.tld to and so on.
    • Sort the files first and then call rsync once for each file: ls -v from/*.tld | xargs -I FROM -n 1 rsync -v from/FROM to This is probably not very efficient.