2 different Benchmarks with 2 different environment variables

Question

Trying to benchmark a function, drawManaged(), running under two different DLLs:

• drawManaged() function in C#
• drawManaged() calls drawNative() in native1base.dll (or other native dlls) through swig-interface
• which native DLL to load is configurable using environmental variables, %NATIVE_DLL_PATH%
• want to benchmark drawManaged() running in 2 different native DLL: native1base.dll and native2optimized.dll

how can I do this using benchmarkdotnet ?

Answer 1

Benchmarkdotnet will benchmark methods marked with the [Benchmark] attribute.

It may be better to create a small utility (or directy in the benchmark project you create) to call this functions. Try and limit all the variables between each call, so instead of setting an environment variable in the native cases and then calling a C# function directly make the [Benchmark] decorated methods call only that which you want to compare.

Your test would look something like this:

    public class DrawManagedVsDrawNative
{
    private DrawManaged drawManaged = new DrawManaged();
    private DrawNative drawNative = new DrawNative();
    private byte[] data;


    [GlobalSetup]
    public void Setup()
    {
       // Some initialization here
    }

    [Benchmark]
    public byte[] DrawManaged() => drawManaged.Draw();

    [Benchmark]
    public byte[] DrawNative() => drawNative.Draw();
}

There are more decorators and configuration options that benchmarkdotnet offers for your test, consult the docs for this

If you really must benchmark setting the environment variables you can create custom jobs using a ManualConfig implementation. Something like this:

 [Config(typeof(ConfigWithCustomEnvVars))]
    public class DrawManagedVsDrawNative
    {
        private class ConfigWithCustomEnvVars : ManualConfig
        {
            private const string EnvVar = "Env_Var_Sample";

            public ConfigWithCustomEnvVars()
            {
                Add(Job.Core.WithId("Variable not set"));
                Add(Job.Core
                    .With(new[] { new EnvironmentVariable(EnvVar, "1") })
                    .WithId("Variable set"));
            }
        }

    private DrawManaged drawManaged = new DrawManaged();
    private DrawNative drawNative = new DrawNative();
    private byte[] data;


    [GlobalSetup]
    public void Setup()
    {
       // Some initialization here
    }

    [Benchmark]
    public byte[] DrawManaged() => drawManaged.Draw();

    [Benchmark]
    public byte[] DrawNative() => drawNative.Draw();

    }