A .Net application I wrote has to read a CSV file storing each record in memory. On my laptop this takes less than a second for the file in question... but since deploying onto a Standard_D11_v2 machine in Azure this part of my application is taking over 15 minutes and I just cannot account for that much of a performance hit. When I ran a quick test to see how long it takes to read a single line (from identical csv files) my laptop is taking 0.014ms while on azure it takes 7.391ms (525x longer!).
I'm using Josh Close's CsvHelper for reading and not doing anything intensive within the read loop. The application is deployed as a cloud service and the Azure machine is pretty bog standard (no premium storage) but this seems like an excessive drop in performance for a fairly simple task. Has anyone else encountered an issue with Azure deployments being very slow when it comes to reading CSV files? Any suggestions as to how I can speed the application up?
Csv read code for reference:
TextReader textReader = new StreamReader(filepath);
CsvReader csv = new CsvReader(textReader);
while (csv.Read())
{
// Get record from file
var record = (IDictionary<string, object>)csv.GetRecord<dynamic>();
if (record != null)
{
// Get 7 fields from record
// create new object from the fields
// store object in dictionary
}
}
Edit - As an aside, the csv file is stored locally on the machine which the application is running on.
The underlying virtual machines for Cloud Services have really low IOPS which seems as though it may have been responsible for causing this issue. The machine I was allocated by cloud services was a D11_V2 Standard which has a max IOPS of 4x500.
I set up my own VM (DS3_V2 Promo which has a max IOPS of 16000) and the csv read now happens in less than a second with no changes to code.