Search code examples
c#visual-studiocloudperformance-testingload-testing

How to use different .csv for each core agent for Visual Studio Cloud Testing?

I'm trying to execute simple REST API load test with Visual Studio Cloud Test: https://www.visualstudio.com/en-us/docs/test/performance-testing/getting-started/getting-started-with-performance-testing

The request contains authorization token and I use pre-generated .csv data source to supply them. Here is the description: https://msdn.microsoft.com/library/ms243142.aspx

When run the test from local machine everything works just fine, but when I start the test from cloud 97% of tests are failing.

Because my service has concurrency check - one token can be used to make only one request. An it seems the test is running from 20 agent machines in cloud and all of them use the same .csv data source.

I wonder if there is a way where I can spread the different data sources per cloud agent?

Solution

  • As per your comment, the data source access method of unique cannot be used with a VSTS CLOUD load test.

    One possibility: Can the tokens you generate include the agent number (AgentID) ? If yes then you might use a plugin containing code based on:

    string tokenFromCSV = e.WebTest.Context["DataSource1.file#csv.token"].ToString();
string agentId = e.WebTest.Context["AgentId"].ToString();
if(agentId.Length==1) agentId = "0" + agentId;
string tokenToUse = tokenFromCSV + agentId;
e.WebTest.Context["tokenToUse"] = tokenToUse;

    Then in the places that currently use the token from the CSV file, use the newly written context parameter tokenToUse. There are many other ways that the agent-id could be merged into the token value.

    Another possibility. If there are never more than 20 (or some other not-too-large number of) agents used then generate a CSV containing 20 columns of token values. Have the column names contain the agent number and then let each agent only use the values from its column. This would probably need a plugin to access the correct column and store the value into a context parameter.

    Another possibility. If the number of lines in the CSV can be several times bigger than the number of virtual users in the test. I am thinking of at least 10 times as many, but the bigger the ratio the better. The set the CSV access method to Random. The tests should then mostly run with tokens only being used by one test at a time. There will be some dual uses and hence test failures but your customer might accept that level of failures.