We collect performance metrics like average response time, latency, Hits etc,. when we do load test. How can we evaluate the performance of a request using these metrics?
When you run performance tests, each test is run with some goal. It could be to baseline the application to assure that the application can handle certain amount of load without any SLA violations. SLAs are defined with product managers to set the performance expectations. These are basically in terms of acceptable response time ranges for APIs, throughput, page load time etc.
When you run your tests for the first time, you are creating a baseline for your application and creating load that is expected in real life. You collect metrics like response time, throughput to make sure that you are meeting SLAs defined for the product or how your application is performing under normal load conditions.
In case build to build performance tests, you need to make sure that throughput for latest and previous build is equal or more and response times have not deteriorated in subsequent builds.
If you running load tests with increasing load levels then response times, throughput can help you understand at what point your system is saturating and cannot scale further and what requests arent scaling well with increased load. In order to achieve this run tests with different load levels and compare response time and throughput for each request for under different load levels. This will give you clear picture of which requests are performing well under load and and which are not. Once you know the poorly performing requests, run tests for that particular request and add code profilers along with the load tests to find the bottlenecks.