Search code examples
visual-studioperformance-testingload-testing

How is 95% response time calculated in visual studio load test

While analyzing VSTS load test report we find response times for an endpoint as below

  • avg response time :- 0.68
  • 90% response time :- 1.18
  • 95% response time :- 1.34
  • 99% response time :- 1.68

The sampling rate is set to 15 sec as recommended by VSTS (When we set below 15 sec we get a warning). We use VSTS cloud offerings to generate load.

In the performance graph (in results) response time peaks shown are at approx 0.7 seconds and nothing above that. When we download the samples we see all entries(1 entry per 15 seconds) have data matching with what is plotted on the graph and is less that 0.7 seconds.

Our SLA is to support 1 sec for 95%. Now we are unable to understand

  1. how are 90%, 95% and 99% response time calculated (with what data)
  2. how can we get more data about the requests which contributed to 95% response time so that we can debug
Solution

  • For each different counter, one value is generated in each sampling interval and that value is shown on the graphs. Hence the graphs often seem to omit peak values. The tables of results collect all the values and the percentiles for response times are calculated by excluding the slowest values. If a total of N response times are collected then the 90th percentile excludes the slowest 10% of those N times, similarly for the other percentiles. To understand the range of response times compare and contrast these values:

    • The percentiles.
    • The average. But remember the average can be distorted by a few very large or very small values.
    • The median. Which is found by excluding almost half of the slowest values and the same number of the fastest values.
    • The minimum and maximum values

    Microsoft have several web pages about analysing load test results. This page is one to start with. A web search for terms such as site:microsoft.com analysing load test results should find much more detail.

    The percentile values are calculated by SQL stored procedures that can be found by exploring the load test database. On my Windows 10 computer it is in the files: C:\Users\AdrianHHH\LoadTest2010.mdf and C:\Users\AdrianHHH\LoadTest2010_log.ldf. The procedures are also shown in the load test database creation script which should be in C:\Program Files\Microsoft Visual Studio 11.0\Common7\IDE\loadtestresultsrepository.sql (but change the 11.0 to reflect your Visual Studio version).