Search code examples
unixrsyncbigdatafile-copying

Rsync performance - syncing a single large file vs syncing multiple small files

I am using rsync to sync many small files (each 4.5 MB). I am using a bwlimit of 18000. I will be executing the bulk rsync command such that,

rsync -v dest_host:file1 dest_host:file2 dest_host:file3 dest_host:file4 dest_host:file5 ... src_dir.

But what I observed is that with the above bandwidth, I am not able to sync more than 4 files per minute. Is this expected? I am wondering if I have synced a single large file of 18 MB, could it be faster?

And for my above use case, are there any utility?

Solution

  • I managed to answer my own question.

    Syncing multiple small files have overhead like file creation, opening, closing both at source and at destination.

    And this will be significant when we transfer 100 files of size 5 MB and one file of size 500M MB.

    So, the verdict, yes syncing a single large file is beneficial.