I was using azcopy to copy models from Azure Blob Storage to Azure VM regularly. But when I am copying datasets to to my VM, I am using Azure File Share and moving data into the data disk using cp command. I want to utilize Azcopy to copy data in parallel. I believe one time I heard that AzCopy copy data in parallel but I am not able to find that statement. Maybe I heard it wrong.
I also saw this another question on stackoverflow which talked about parallelism in azcopy. The answer provided a link to the azcopy docuemntation and talked about --parallel-level but when I clicked on it, there is no such thing as it has been stated.
If anyone can redirect me to the azcopy parallel documentation link if it exists it would be really helpful.
AzCopy copies data in parallel by default, but you can change how many files are copied in parallel.
Throughput can decrease when transferring small files. You can increase throughput by setting the AZCOPY_CONCURRENCY_VALUE environment variable. This variable specifies the number of concurrent requests that can occur.
If your computer has fewer than 5 CPUs, then the value of this variable is set to 32. Otherwise, the default value is equal to 16 multiplied by the number of CPUs. The maximum default value of this variable is 3000, but you can manually set this value higher or lower.