BenchmarkDotNet Unable to find Tests when it faces weird solution structure

Question

I have problem with BenchmarkDotNet which I struggle to solve

Here's my project structure:

- 
    | - Infrastructure
    |        |
    |        | - TestsBenchmark
    |        | - MyInfra.sln
    | - src
            | - Tests
            | - MyProduct.sln

TestsBenchmark references Tests and has only this line of code:

BenchmarkSwitcher.FromAssembly(typeof(BasicTests).Assembly).RunAll();

but when I run it via dotnet run -c Release it throws

// Generate Exception: Unable to find Tests in ...\Infrastructure and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

Previously when my project structure was like this:

- 
    | - src
            | - Tests
            | - TestsBenchmark

everything worked fine

Reproduction steps (manual), it recreates folders structure, projects, projects relations, solution and add nugets. run it in e.g powershell in some empty folder:

mkdir Infrastructure
mkdir src
cd src
dotnet new xunit -n Tests
cd Tests
dotnet add package BenchmarkDotNet
cd ..
cd ..
cd Infrastructure
dotnet new console -n TestsBenchmark
cd TestsBenchmark
dotnet add package BenchmarkDotNet
cd ..
dotnet new sln -n Repro
dotnet sln add .\TestsBenchmark\TestsBenchmark.csproj
dotnet sln add .\..\src\Tests\Tests.csproj
cd TestsBenchmark
dotnet add reference "..\..\src\Tests\Tests.csproj"

UnitTest1.cs

using System;
using BenchmarkDotNet.Attributes;
using Xunit;

namespace Tests
{
    public class UnitTest1
    {
        [Fact]
        [Benchmark]
        public void Test1()
        {
            Console.WriteLine("asd");
        }
    }
}

Program.cs

using System;
using BenchmarkDotNet.Running;
using Tests;

namespace TestsBenchmark
{
    class Program
    {
        static void Main(string[] args)
        {
            BenchmarkSwitcher.FromAssembly(typeof(UnitTest1).Assembly).RunAll();
        }
    }
}

and now inside Infrastructure\TestsBenchmark

perform dotnet run -c Release

and you'll see

// Generate Exception: Unable to find Tests in C:\\Infrastructure and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// BenchmarkDotNet has failed to build the auto-generated boilerplate code.
// It can be found in C:\\repro\Infrastructure\TestsBenchmark\bin\Release\net5.0\65ba2c51-e794-4f44-93ab-f811411c86f5
// Please follow the troubleshooting guide: https://benchmarkdotnet.org/articles/guides/troubleshooting.html

Answer 1

The short answer is you cannot run benchmark with the structure you created and it is intentional.

For the BenchmarkDotNet (and it is a generally good practice) it's required for solution to have following structure

   | - root
             |
             | - Project1/Project1.csproj
             | - Project2/Project2.csproj
             | - Project3/Project3.csproj
             | - ProjectInsideFolder
                       | - Project4/Project4.csproj
             | - YouSolution.sln

So you can put projects in sub-folders or virtual sub-folders, but the sln file must be in the root directory.

BenchmarkDotNet needs to find the csproj file for you test project and to do that it uses following algorithm

Then it searches for the project file in all subdirectories and the root directory itself.

Note on using same project in 2 or more solutions: Don't use the same project in 2 different solutions unless you absolutely have to. It has its consequences and there is also an option to reuse it as nuget-package you can create your own nuget feed for that.