JMH Multiple Benchmarks Different results on isolated runs

Question

I run some benchmarks through JMH

Supposing I have three benchmarks listed in one Java File

    @Benchmark
    @Threads(value = 2)
    @Warmup(iterations = 10)
    @Fork(value = 2)
    @Measurement(iterations = 10)
    public String benchmarkA() {
        ...
    }

    @Benchmark
    @Threads(value = 2)
    @Warmup(iterations = 10)
    @Fork(value = 2)
    @Measurement(iterations = 10)
    public String benchmarkB() {
       ...
    }

    @Benchmark
    @Threads(value = 2)
    @Warmup(iterations = 10)
    @Fork(value = 2)
    @Measurement(iterations = 10)
    public String benchmarkC() {
        ...
    }

Each method benchmarks a different algorithm that serves the same purpose. When I run all the benchmarks together, results vary in comparison to running the benchmarks one by one.

For example execute a JMH run with only onw benchmark on the Java file.

    @Benchmark
    @Threads(value = 2)
    @Warmup(iterations = 10)
    @Fork(value = 2)
    @Measurement(iterations = 10)
    public String benchmarkA() {
        ...
    }

The results for this method would be better If only this method runs, however in the case of more benchmarks the results would be worse.

Supposing result wise benchmarkA > benchmarkB > benchmarkC by running them isolated, if I run them all in one go the differences between them get amplified.

Also I tried changing the order of the benchmarks (executed in lexicographic order) and the results are the same. It is as if the number of benchmarks affects the results.

What could be a reason?

Answer 1

Each benchmark is executed in its own JVM (unless explicitly set to behave otherwise), so cross benchmark pollution should not happen. What can happen, and is quite common, is the thermal throttling of your CPU. If you are benchmarking in an environment where you have no control over CPU frequency the typical behaviour would be for the first benchmark to push CPU frequency up, and later benchmarks to get throttled. You can control for this variable in Linux environments by switching to a user controlled power governor and setting a fixed frequency for your CPU.