Search code examples
c#benchmarkingfeedback

C#: Is this benchmarking class accurate?


I created a simple class to benchmark some methods of mine. But is it accurate? I am kind of new to benchmarking, timing, et cetera, so thought I could ask for some feedback here. Also, if it is good, maybe somebody else can make use of it as well :)

public static class Benchmark
{
    public static IEnumerable<long> This(Action subject)
    {
        var watch = new Stopwatch();
        while (true)
        {
            watch.Reset();
            watch.Start();
            subject();
            watch.Stop();
            yield return watch.ElapsedTicks;
        }
    }
}

You can use it like this:

var avg = Benchmark.This(() => SomeMethod()).Take(500).Average();

Any feedback? Does it look to be pretty stable and accurate, or have I missed something?


Solution

  • It is about as accurate as you can get for a simple benchmark. But there are some factors not under your control:

    • load on the system from other processes
    • state of the heap before/during the benchmark

    You could do something about that last point, a benchmark is one of the rare situations where calling GC.Collect can be defended. And you might call subject once beforehand to eliminate any JIT issues. But that requires calls to subject to be independent.

    public static IEnumerable<TimeSpan> This(Action subject)
    {
        subject();     // warm up
        GC.Collect();  // compact Heap
        GC.WaitForPendingFinalizers(); // and wait for the finalizer queue to empty
    
        var watch = new Stopwatch();
        while (true)
        {
            watch.Reset();
            watch.Start();
            subject();
            watch.Stop();
            yield return watch.Elapsed;  // TimeSpan
        }
    }
    

    For bonus, your class should check the System.Diagnostics.Stopwatch.IsHighResolution field. If it is off, you only have a very coarse (20 ms) resolution.

    But on an ordinary PC, with many services running in the background, it is never going to be very accurate.