Using TestCaseFilter with VSTest and Native Unit Tests

Question

I am unable to run vstest with a /TestCaseFilter that excludes certain tests by category. If I use the "Test Explorer" in Visual Studio 2017 and filter by "Traits" it is working correctly but ultimately I want to be able to edit the TFS build definition and exclude certain tests from running on the build server.

I used the example from the following to add a "TestCategory" attribute to certain tests: http://www.ademiller.com/blogs/tech/2014/04/test-categories-for-visual-c/

I've tried various versions of the command line but cannot get vstest to recognize my "TestCategory" filter:

https://blogs.msdn.microsoft.com/vikramagrawal/2012/07/23/running-selective-unit-tests-in-vs-2012-rc-using-testcasefilter/

https://github.com/Microsoft/vstest-docs/blob/master/docs/filter.md

https://learn.microsoft.com/en-us/visualstudio/test/vstest-console-options?view=vs-2019

vstest.console.exe /TestCaseFilter:"TestCategory=Slow" "D:\MyStuff\Project.Tests.dll" /Platform:x64 /InIsolation'

I get the following error: No test matches the given testcase filter TestCategory=Slow in D:\MyStuff\Project.Tests.dll"

I have tried escape characters, different types of quotes, parentheses, but nothing works. If I set the filter to be "TestCategory!=Slow", all tests will execute, even the slow ones!

Any help is appreciated. Thanks.

Answer 1

It's been a little over two years but this is finally working.

In a header file, I declare the following:

#define EXPENSIVE_TEST_METHOD(name) \
  BEGIN_TEST_METHOD_ATTRIBUTE(name) \
     TEST_OWNER(L"Expensive") \
  END_TEST_METHOD_ATTRIBUTE() \
  TEST_METHOD(name)

I can then use EXPENSIVE_TEST_METHOD in place of TEST_METHOD in the unit test.

We're now using Azure DevOps Server 2019 so in the Visual Studio Test task, I can now add the following test filter on CI builds:

On command line, you can now do something like this:

vstest.console.exe "%MyTestProjectDll%" /Platform:x64 /InIsolation /TestCaseFilter:"Owner!=Expensive"