How to get per scenario reports from parameterized simulations in gatling?

Question

I'm trying to test a single API that internally does different things based on inputs:

• country
• customer
• amount of items

The following simulation is what I came up with:

val countries = List("US", "CAN")
val customerTypes = List("TYPE1", "TYPE2")
val basketSizes = List(1, 10, 50)

val scenarioGenerator: Seq[(String, String, Int)] = for {
  country <- countries
  customerType <- customerTypes
  basketSize <- basketSizes
} yield (country, customerType, basketSize)

def scenarios(): Seq[PopulationBuilder] = {
  var scenarioList = new ArraySeq[PopulationBuilder](countries.size * customerTypes.size * basketSizes.size)
  var i = 0;
  for ((country: String, customerType: String, basketSize: Int) <- scenarioGenerator) {
    // fetch customer data for scenario
    val customers = DataFetcher.customerRequest(country, customerType)
    // fetch product data for scenario
    val products = DataFetcher.productRequest(country)

    // generate a scenario with given data and parameters
    val scen = scenario(s"Pricing-(${country},${customerType},${basketSize})")
      // feeder that creates the request object for the gatling user
      .feed(new PricingFeeder(country, customers, products, basketSize))
      .repeat(10) {
        exec(Pricing.price)
          .pause(500 milliseconds)
      }
      .inject(
        rampUsers(10) over (10 seconds)
      )
    scenarioList(i) = scen

    i = i + 1
  }

  scenarioList
}

setUp(scenarios: _*).protocols(httpProto)

This is run with the maven plugin (and with tracking in jenkins using the gatling plugin), but this results in a single tracked case: Pricing. This is useless as even the item amount will be close to a linear increase in response time.

The simulation.log has the data for each scenario type, but out of the box reporting handles it as a single type of query, and merges all the results in a single graph, which means that it's impossible to see if a certain combination causes a spike due to a calculation or data bug.

I'd like to get separate metrics for each of the combinations, so it would be really easy to see for example that a code or data change in the API caused a latency spike in the Pricing-(US,TYPE1,50) scenario.

What is the idiomatic way of achieving this with gatling? I don't want to create simulations for each case, as this would be a nightmare to manage (and getting rid of manually managed data and jenkins jobs with jmeter is what we are trying to achieve).

Answer 1

First thing - it is not a good practice to run so many scenarios in one simulation as it runs them in parallel not sequentially so you should be sure that it is what you want.

If so, you can use fact that gatling report allows to show graphs per group. So you can wrap all your requests in group that is named based on parameters, this way in detailed view of report you will be able to select which group to show, fe.:

 val singleScenario = scenario(s"Pricing-(${country},${customerType},${basketSize})")
   .group(s"Pricing-(${country},${customerType},${basketSize})"){
      .feed(new PricingFeeder(country, customers, products, basketSize))
        .repeat(10) {
          exec(Pricing.price)
            .pause(500 milliseconds)
        }
    }

If you do not need all scenarios to run in parallel, and want separate reports for separate scenarios best way is to implement simulation class as parametrized abstract class and add separate classes for each parameters set, as in Gatling one simulation equals on report, fe.:

package com.performance.project.simulations

import io.gatling.core.Predef.Simulation
import scala.concurrent.duration._

class UsType1Simulation1 extends ParametrizedSimulation("US", "TYPE1", 1)
class UsType1Simulation10 extends ParametrizedSimulation("US", "TYPE1", 10)
class UsType1Simulation50 extends ParametrizedSimulation("US", "TYPE1", 50)

class UsType2Simulation1 extends ParametrizedSimulation("US", "TYPE2", 1)
class UsType2Simulation10 extends ParametrizedSimulation("US", "TYPE2", 10)
class UsType2Simulation50 extends ParametrizedSimulation("US", "TYPE2", 50)

class CanType1Simulation1 extends ParametrizedSimulation("CAN", "TYPE1", 1)
class CanType1Simulation10 extends ParametrizedSimulation("CAN", "TYPE1", 10)
class CanType1Simulation50 extends ParametrizedSimulation("CAN", "TYPE1", 50)

class CanType2Simulation1 extends ParametrizedSimulation("CAN", "TYPE2", 1)
class CanType2Simulation10 extends ParametrizedSimulation("CAN", "TYPE2", 10)
class CanType2Simulation50 extends ParametrizedSimulation("CAN", "TYPE2", 50)

sealed abstract class ParametrizedSimulation(country: String, customerType: String, basketSize: Int) extends Simulation{

  val customers = DataFetcher.customerRequest(country, customerType)
  val products = DataFetcher.productRequest(country)

  val singleScenario = scenario(s"Pricing-(${country},${customerType},${basketSize})")
    .feed(new PricingFeeder(country, customers, products, basketSize))
    .repeat(10) {
      exec(Pricing.price)
      .pause(500 milliseconds)
    }
    .inject(
      rampUsers(10) over (10 seconds)
    )

  setUp(singleScenario).protocols(httpProto)
}

Of course it makes sense only if there is small amount of combinations, with hundreds of them it will get messy.