What is the best solution to execute a load test to simulate 200k - 500k concurrent users?

Question

I'm looking for the best solution to execute a load test to simulate 200k - 500k concurrent users to test the web app. Some scenarios have to do with video viewing, user content pushing (like pictures, video uploading), page viewing. One of the goals is just to make sure that CDN service can handle such a huge load from the user perspective.

From the research, it seems the Jmeter can be distributed as described here:

• docker https://hub.docker.com/r/thothbot/jmeter-master/
• AWS + docker https://dragoscampean.medium.com/how-to-build-a-distributed-load-testing-infrastructure-with-aws-docker-and-jmeter-accf3c2aa3a3

Questions:

1. But, how to estimate how many slave nodes will be needed and with which hardware when looking for the cloud?

2. What about BlazeMeter service? It looks like this service can save time - just avoid infrastructure setups.

Actually, without an answer to question 1, it's not possible to calculate the cost and compare.

Any suggestion? The idea to make the target test in a strict time frame, using the best smart solution :)

Answer 1

Docker per se doesn't solve the scaling challenge, it does the opposite thing: creates an overhead which consumes resources. So unless you have a container orchestration solution like Docker Swarm or K8S which can automatically scale or create new containers on physical or virtual machines - it won't give you any performance benefit comparing to running JMeter in distributed mode on bare metal or virtual machines without using containers.

1. There is no answer like "c6g.large AWS instance can simulate 5000 virtual users" because the actual number depends on:

   • machine hardware specifications
   • nature of your test (nature of samplers, pre/post processors, assertions, etc.)
   • application response time and size

   so if you're executing simple HTTP GET requests which return small amount of data you will be able to simulate more users than in case of uploading large files and receiving large responses

   So you should act as follows:

   • Make sure to have a solution to monitor the health of JMeter engine(s) in terms of CPU, RAM, Network, Disk IO, etc. If you don't have one in mind you can consider using JMeter PerfMon Plugin

   • Make sure to follow JMeter Best Practices

   • Start with 1 virtual user and gradually increase the load at the same time looking at resources consumption

   • When any of monitored resources consumption start exceeding reasonable threshold, i.e. 80% of total available capacity - take a note how many users where online at this stage by looking at i.e. Active Threads Over Time listener

   • This is how many users you can simulate from particular this machine for particular this test

2. BlazeMeter engines hardware capacities are also limited and the above constraints apply so make sure to monitor engines health during your load test as if JMeter is not capable of sending requests fast enough you will get "false negative" results even if your application can process more requests per unit of time