I'm running some startup scripts (cmd/bat) on my small azure VM which include a file-transfer operation from a mounted VHD, and normally it finishes in about 3 minutes (copying files and extracting ~500Mb zip file with command-line 7z).
When I scale out to ~150 instances, the same operation is very slow (up to 15 minutes in total, most of which is used by 7z). Also, the nodes which are the slowest to complete the bootup procedure are very hard to access at first using mstsc(animation is lagging and takes a lot of time to log in), but that might not be related.
What could be the problem?
We had the idea to examine the cache, but it would be nice to know of any other potential bottleneck which may be present in the following situation.
UPDATE: I tried extracting on the D:\ drive instead of extracting it on the C:\ and while scaling to 200, the unzip takes about a minute! So it seems like the problem is that C:\ might be on the blob. But again, I have 3GB of data in 40 files, so 60MB/s per blob should be enough to handle it. Or - can it be the case that we have a cap for the all blobs?
one thing to consider is the per-storage-account scalability target of a storage account. With georeplication enabled, you have 10Gbps egress and 20K transactions/sec, which you could be bumping into. Figure with 150 instances, you could potentially be pulling 150 x 100Mbps, or 15Gbps as all of your instances are starting up.
Not sure about the "mounted VHD" part of your question. With Azure's drive-mounting, only one virtual machine instance can mount to a drive at any given time. For this type of file-copy operation, typically you'd grab a file directly from a storage blob, rather than a file stored in a vhd (which, in turn, is stored in a page blob).
EDIT: Just wanted to mention that an individual blob is limited to 60MB/sec (also mentioned in the blog post I referenced). This could also be related to your throttling.