How can I create a custom filter for NUnit 3 programmatically, with ITestFilterService and/or ITestFilterBuilder

TLDR: how to programmatically filter NUnit 3.x tests based on partial testname, regex or (partial) category names.

I'm converting a project from that has its own filter mechanism for running tests from NUnit 2.6 -> NUnit 3.8.

The filters I am trying to support are partial string match on the name, regex match, name match and category match. Since this project has some 30k+ tests, some of them time-consuming, a custom runner has proven helpful to keep us sane.

However, I seem to not be able to see the forest for the trees. I've tried to read the docs on Test Engine API, which essentially links to the source code of the interfaces. It seems to me I need and ITestFilterService and from that the ITestFilterBuilder, but then I seem to get stuck.

NUnit 2.6 came with a bunch of predefined filters which I only needed to extend. Now, I sympathize with the new approach in decoupling the test runner API from the NUnit framework, and I've no problem diving in the docs, but it seems that they are sparse (or my searching skills, are lacking).

I got as far as the following (F# code, translated from C# code), but then I got stuck:

let path = Assembly.GetExecutingAssembly().Location
let package = new TestPackage(path)
package.AddSetting("WorkDirectory", Environment.CurrentDirectory)
let engine = TestEngineActivator.CreateInstance()
let filterService = engine.Services.GetService<ITestFilterService>()
let filterBuilder = filterService.GetTestFilterBuilder()

I don't even know if it works, as I cannot run it until I refactor the whole project... The ITestFilterBuilder has three methods, AddTest, GetFilter and SelectWhere. But these take strings and I have no clue what to put into them.

It seems like I need SelectWhere, which takes a whereClause, but no help in the tooltips, intellisense or anywhere on what syntax to use (SQL? XPath?).

Anybody any ideas? Some project out there that has some predefined filters I can use, or example code out there?

Solution

You can get an idea of what to add to the filters if you pass an empty one in like so:

var filterService = nunitEngine.Services.GetService<ITestFilterService>();
ITestFilterBuilder builder = filterService.GetTestFilterBuilder();
....
var testResult = runner.Run(testListener, filter);

When you run this, assuming all is ok, you'll get an XMLNode response with some nested <test-suite> nodes, under which there will be some <test-case> nodes:

<test-suite 
    type="TestFixture" 
    id="0-1000" 
    name="Tests" 
    fullname="SampleUnderTest.Tests.NUnit.Tests" 
    classname="SampleUnderTest.Tests.NUnit.Tests" 
    runstate="Runnable" 
    testcasecount="1" 
    result="Passed" 
    start-time="2020-08-26 15:51:23Z" 
    end-time="2020-08-26 15:51:23Z" 
    duration="0.031570" 
    total="1" 
    passed="1" 
    failed="0" 
    warnings="0" 
    inconclusive="0" 
    skipped="0" 
    asserts="1">
  <test-case 
       id="0-1001" 
       name="AddWithGivenInputsReturnsExpectedResults"
       fullname="SampleUnderTest.Tests.NUnit.Tests.AddWithGivenInputsReturnsExpectedResults"            
       methodname="AddWithGivenInputsReturnsExpectedResults" 
       classname="SampleUnderTest.Tests.NUnit.Tests" 
       runstate="Runnable" 
       seed="81337978" 
       result="Passed" 
       start-time="2020-08-26 15:51:23Z" 
       end-time="2020-08-26 15:51:23Z" duration="0.028935" asserts="1" />
</test-suite>

Now, back to our ITestFilterBuilder:

ITestFilterBuilder builder = filterService.GetTestFilterBuilder();

One of the public methods of this is ITestFilterBuilder.AddTest(string fullName)

The parameter fullName matches the fullName attribute of the XML returned. if we pass in the fullName from one of out <test-case> nodes, we are able to run just that test:

builder.AddTest("SampleUnderTest.Tests.NUnit.Tests.AddWithGivenInputsReturnsExpectedResults");

When we run again, the result the results on our root <test-suite> tell us how many tests we've run, passed, failed, etc and various other stats about it.