What would be the best way to add Performance Testing in Continuous Integration/Delivery Environment

Question

I'm about to integrate performance testing on CI, but I'm having a hard time to decide which ones I should execute and how to run it:

1. Should I run load testing on individual APIs or should I run the whole workflow (e.g.: Login -> Home Page -> Search)
2. How long I should run it?
3. Should I also add stress, peak and soak testing?
   I'm thinking, those should be run out of the CI.

Any comments would be really appreciated.

Answer 1

The main idea of including tests in continuous integration process is protecting yourself from regressions, i.e. to ensure that a new feature or a bug fix doesn't cause performance degradation.

So common practice is to have short-term Load Test covering essential features and conducting anticipated number of users to be run periodically (i.e. on each commit or pull request to master/integration branch).

If you have enough capacity and environment availability it would also make sense to have soak test in place, but it should occur less frequently (you want to keep build time short enough, don't you?), i.e. overnight or over the weekend. This way you will be able to identify possible memory leaks

Both approaches assume having some reference metrics collected by previous ad-hoc runs which can be considered as acceptable pass/fail criteria.

Stress test assumes identifying saturation/breaking points so normally it is not included into CI pipeline and being run manually before major releases.

Check out How to Include Load Testing in your Continuous Integration Environment article for more information.