How exactly does JMH do the calculations?Where in the code or in which file can you see the process and formula for calculating "score" and "error"?

Question

How exactly does JMH do the calculations? Where in the code or in which file can you see the process and formula for calculating "score" and "error"?

CopyOnWriteBenchmark.testAdd       avgt    5  223968,116 ± 553800,476  us/op
CopyOnWriteBenchmark.testAddAt     avgt    5     198,910 ±     98,401  us/op
CopyOnWriteBenchmark.testContains  avgt    5      14,969 ±      1,080  us/op
CopyOnWriteBenchmark.testGet       avgt    5       0,004 ±      0,001  us/op
CopyOnWriteBenchmark.testIndexOf   avgt    5      20,646 ±      3,313  us/op
CopyOnWriteBenchmark.testRemove    avgt    5      16,443 ±      2,015  us/op```

Answer 1

looking into debugger I see that the code responsible for calculations is generated at compilation time separately for each benchmark. Below there is an example of generated method:

public static void dto_avgt_jmhStub(InfraControl control, RawResults result, BenchmarkParams benchmarkParams, IterationParams iterationParams, ThreadParams threadParams, Blackhole blackhole, Control notifyControl, int startRndMask, ProjectionVsDtoBenchmark_jmhType l_projectionvsdtobenchmark0_0) throws Throwable {
    long operations = 0;
    long realTime = 0;
    result.startTime = System.nanoTime();
    do {
        blackhole.consume(l_projectionvsdtobenchmark0_0.dto());
        operations++;
    } while(!control.isDone);
    result.stopTime = System.nanoTime();
    result.realTime = realTime;
    result.measuredOps = operations;
}

Here l_projectionvsdtobenchmark0_0.dto() is the method annotated with @Benchmark. The method above is in turn called from this one:

public BenchmarkTaskResult dto_AverageTime(InfraControl control, ThreadParams threadParams) throws Throwable {
    this.benchmarkParams = control.benchmarkParams;
    this.iterationParams = control.iterationParams;
    this.threadParams    = threadParams;
    this.notifyControl   = control.notifyControl;
    if (this.blackhole == null) {
        this.blackhole = new Blackhole("Today's password is swordfish. I understand instantiating Blackholes directly is dangerous.");
    }
    if (threadParams.getSubgroupIndex() == 0) {
        RawResults res = new RawResults();
        ProjectionVsDtoBenchmark_jmhType l_projectionvsdtobenchmark0_0 = _jmh_tryInit_f_projectionvsdtobenchmark0_0(control);

        control.preSetup();


        control.announceWarmupReady();
        while (control.warmupShouldWait) {
            blackhole.consume(l_projectionvsdtobenchmark0_0.dto());
            res.allOps++;
        }

        notifyControl.startMeasurement = true;
        dto_avgt_jmhStub(control, res, benchmarkParams, iterationParams, threadParams, blackhole, notifyControl, startRndMask, l_projectionvsdtobenchmark0_0);
        notifyControl.stopMeasurement = true;
        control.announceWarmdownReady();
        try {
            while (control.warmdownShouldWait) {
                blackhole.consume(l_projectionvsdtobenchmark0_0.dto());
                res.allOps++;
            }
        } catch (Throwable e) {
            if (!(e instanceof InterruptedException)) throw e;
        }
        control.preTearDown();

        if (control.isLastIteration()) {
            f_projectionvsdtobenchmark0_0 = null;
        }
        res.allOps += res.measuredOps;
        int batchSize = iterationParams.getBatchSize();
        int opsPerInv = benchmarkParams.getOpsPerInvocation();
        res.allOps *= opsPerInv;
        res.allOps /= batchSize;
        res.measuredOps *= opsPerInv;
        res.measuredOps /= batchSize;
        BenchmarkTaskResult results = new BenchmarkTaskResult((long)res.allOps, (long)res.measuredOps);
        results.add(new AverageTimeResult(ResultRole.PRIMARY, "dto", res.measuredOps, res.getTime(), benchmarkParams.getTimeUnit()));
        this.blackhole.evaporate("Yes, I am Stephen Hawking, and know a thing or two about black holes.");
        return results;
    } else
        throw new IllegalStateException("Harness failed to distribute threads among groups properly");
}

The results are collected into BenchmarkTaskResult, which in turn contains collection of Result objects, in this case AverageTimeResult containing Statistics field.