Search code examples
filefile-transfertransfer

Why is data transfer speed slower for multiple smaller files than a huge file?


I have always noticed that when for example a react project or a whole website for an instance whose size sums up to say 350-450 MB takes a longer time to transfer than a 2-3 GB video file or for the matter of fact any number of files that are bundled into one file like a .zip/.rar/.iso.

Some SCs :-

Transferring some react projects:- Transferring some react projects

Transferring a movie:- Transferring a movie

I scoured the web but unfortunately found no relevant posts/answers.
Maybe I didn't use the necessary keywords...idk

If possible, a detailed explanation would be really helpful :)
Even if the explanation dives into OS concepts it's cool. I just wanna know why


Solution

  • There is actually a lot going on behind the 'scenes' when you transfer a file:

    • Opening the original file
      • Map the path string to something the operating system actually understands
      • Security is checked (FILE_ACCESS): are you allowed to read and delete the file?
      • More security is checked (FILE_SHARE): is you are allowed, is anyone else doing something with the file?
    • Creating the new file
      • Map the path string to something the operating system actually understands
      • Again security (FILE_ACCESS): are you allowed to create a file here? And the directory above it, and above that...
      • Create the actual new file, and put it into the file system lookup table
      • Set the default file attributes
      • Disallow others to read/write/delete the new file (FILE_SHARE)
    • The actual transfer
      • Read 4k bytes
      • Write 4k bytes, and repeat
    • The original file is removed
    • Set last modified date of the new file
    • The new file is closed

    I'm sure I'm already missing a ton of steps here, but I'm here to illustrate a lot of stuff is going on before and after the actual transfer. Even if the file is empty (0 bytes) we still have to do all this. That's apart from the already given answers such as fragmentation.