Search code examples
visual-studiotestingvisual-studio-2017performance-testingload-testing

Which test mix model option to select for REST Api Load Testing in Visual Studio?

I have created the WebTest file and now adding new Load Test using the WebTest file. The WebTest file contains api snapshot of my session that performs several REST Apis including Post, Get, etc. The Web Performance test is passing successfully.

I am creating the Load Test using Load Test Wizard in Visual Studio where I am asked to pick up Test Mix Model as shown in screenshot:

Screenshot1

There are following test mix model options for your load test scenario:

  1. Based on the total number of tests

  2. Based on the number of virtual users

  3. Based on user pace

  4. Based on sequential order

I referred the official documentation but I am not able to understand still which model mix to use and what is the difference and performance impact each mix create?

Solution

  • As you only have one web test then I would suggest using the test mix based on number of tests. But actually any mix except user pace should be suitable.

    The four test mixes behave as described below. I recommend careful reading of the words shown in the screenshot in the question, and the related words for the other mixes. The diagram above those words is intended to suggest how the mixes work.

    As there is only one web test, the "sequential order" mix is not useful. It is only used when you have more than one test (e.g., A, B and C) and they need to be run repeatedly in that order (i.e. A B C A B C A B C ... A B C) by each virtual user (VU).

    The "user pace" mix is used when you want a web test to be run a fixed rate (of N times per hour) to apply a steady load to the system under test. Suppose test D takes about 40 seconds to execute and the pace is set to 30 tests per hour. Then we expect the system to start, for each VU, that test every 2 minutes (and hence get about 100 seconds idle between executions). Suppose now that there is also test E that takes about 15 seconds to execute and its pace is set to 90 per hour. Then each VU should additionally run test E every 40 seconds. So now each VU runs tests D and E at those rates. Test D wants 30 * 40 seconds per hour of each VUs time and test E wants 90 * 15 seconds, thus we have 30*40+90*15 = 2550 seconds used of each hour. So each VU has plenty of time to run both tests. Should the tests take much longer or require a faster rate then that might take over 3600 seconds per hour. That is impossible and I would expect to see an error or warning message about it as the test runs.

    The other two test mixes are used when you want to run web tests as quickly as possible (within the number of VUs specified and allowing for think times). The distinction is how web tests are chosen to be run. When there is only one web test then the two mixes have the same effect. With more than one web test one mix tries to make the number of web tests executed match the ratios (percentages) specified in the mix. The other mix tries to make the number VUs running each test match the ratios.

    Suppose we have tests: A at 40%, B at 30%, C at 20% and D at 10%. With a test mix based on total number of tests we would expect the number of each test executed to approximate the stated percentages. Suppose we have 20 VUs and a test mix based on number of users then we would expect 8 VUs to run test A, 6 VUs to run test B, 4 VUs to run test C and 2 VUs to run test D.