How to measure average cold start time with JMH?

Question

In JMH(Java Microbenchmark Harness), we can use

@BenchmarkMode(Mode.AverageTime)
@Warmup(iterations = 10)
@Measurement(iterations = 10)

to evaluate the average time of an execution after JVM warms up.

Also we can use

@BenchmarkMode(Mode.SingleShotTime)
@Measurement(iterations = 1)

to estimate the cold start time of an execution. But this only executes the benchmark once, which may introduce bias. So is there any method to evaluate the average time of the cold start in JMH?

Answer 1

According to Alexey himself (though from 2014):

Single-shot benchmarks were originally destined to run a single measurement iteration over multiple forks -- the scenarios to estimate "cold" performance. But for many cases, you might want more measurement iterations there especially if you are running only a single fork, because more samples would be generated.

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class AverageSingleShot {

    public static void main(String[] args) throws Exception {
        Options opt = new OptionsBuilder()
            .include(AverageSingleShot.class.getSimpleName())
            .build();

        new Runner(opt).run();
    }

    @Fork(100)
    @Benchmark
    @BenchmarkMode(Mode.SingleShotTime)
    public int test() {
        return ThreadLocalRandom.current().nextInt() + ThreadLocalRandom.current().nextInt();
    }

}

Besides the fact that this will tell you the average (see that 100):

 Benchmark               Mode  Cnt      Score      Error  Units
 AverageSingleShot.test    ss  100  41173.540 ± 2871.546  ns/op
 

you will also get Percentiles and a Histogram.