BenchmarkDotNet running same benchmark multiple times

Question

Faced with the issue that my current configuration for the benchmarks did not working properly, the reason of it is that I'm trying to run the benchmark only ones, for that as it mentioned in documentation I've to use this attribute [SimpleJob(RunStrategy.ColdStart, targetCount: 1)] and this was getting me in the wrong direction because from the console I've notice that my single bench was founded twice.

// ***** BenchmarkRunner: Start   *****
// ***** Found 2 benchmark(s) in total *****
// ***** Building 2 exe(s) in Parallel: Start   *****

internal class Program
{
    static async Task Main(string[] args) => BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args, new DebugInProcessConfig());
}

[MarkdownExporter]
[AsciiDocExporter]
[HtmlExporter]
[CsvExporter]
[RPlotExporter]
[SimpleJob(RunStrategy.ColdStart, targetCount: 1)]
public class MyBench
{
    [Params(2)] public int _anotherValueToTestWith;

    [Params(2)] public int _valueToTestWith;

    [GlobalSetup]
    public void GlobalSetup()
    {
        // ...
    }

    [GlobalCleanup]
    public void GlobalCleanup()
    {
        // CleanUp
    }

    [Benchmark]
    public void AccessTokenServiceBench()
    {
        // Perform bench
    }
}

What I'm missing here?

Answer 1

The DebugInProcessConfig defines one Job:

public override IEnumerable<Job> GetJobs()
    => new[]
    {
        Job.Default

and you add one more using the attribute:

[SimpleJob(RunStrategy.ColdStart, targetCount: 1)]

so the resulting config ends up having two jobs. And BDN runs every benchmark for every job defined.

The easiest fix would be to define your own config that runs the benchmark once (Job.Dry) with the desired toolchain:

public class DebugInProcessConfigDry : DebugConfig
{
    public override IEnumerable<Job> GetJobs()
        => new[]
        {
            Job.Dry // Job.Dry instead of Job.Default
                .WithToolchain(
                    new InProcessEmitToolchain(
                        TimeSpan.FromHours(1), // 1h should be enough to debug the benchmark
                        true))
        };
}