xunit from TFS2015 - where to put ParallelizeAssemblies

I'm trying to speed up our unit tests. By setting xunit.parallelizeAssembly to true in in the app.config files I get multiple tests from different assemblies to run in parallel when run from Visual Studio. But when running on the build server it makes no difference in execution time and I can see than only one core is used.

In the paragraph on MSBuild Runner on this page it is suggested that the setting ParallelizeAssemblies would solve this problem. I'm currently running the tests with the "Visual Studio Test" build step (see image for configuration). Where do I put this setting?

I can't share all of the log but I believe the first and last part might contain good clues.

2017-04-20T16:51:10.5496891Z Executing the powershell script: C:\Tfs_Agent5\tasks\VSTest\1.0.32\VSTest.ps1
2017-04-20T16:51:12.9402898Z ##[debug]Calling Invoke-VSTest for all test assemblies
2017-04-20T16:51:12.9559206Z ##[warning]Install Visual Studio 2015 Update 1 or higher on your build agent machine to run the tests in parallel.
2017-04-20T16:51:13.0027923Z Working folder: C:\Tfs_Agent5\_work\1
2017-04-20T16:51:13.0027923Z Executing C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "C:\Tfs_Agent5\_work\1\s\src\AwtSG.CompareToAros\bin\Release\AwtSG.Domain.MetOceanData.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.Dynamics.UnitTests\bin\Release\AwtSG.Domain.Dynamics.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.EcdisRouteFiles.UnitTests\bin\Release\AwtSG.Domain.EcdisRouteFiles.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.EcdisRouteFiles.UnitTests\bin\Release\AwtSG.Domain.NavigationUtilities.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.GeoSpatial.UnitTests\bin\Release\AwtSG.Domain.Geospatial.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.GeoSpatial.UnitTests\bin\Release\AwtSG.Domain.NavigationUtilities.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.MetOceanData.GribApi.UnitTests\bin\Release\AwtSG.Domain.MetOceanData.GribApi.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.MetOceanData.UnitTests\bin\Release\AwtSG.Domain.MetOceanData.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.NavigationUtilities.UnitTests\bin\Release\AwtSG.Domain.NavigationUtilities.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.Optimization.Evolutionary.IntegrationTests\bin\Release\AwtSG.Domain.MetOceanData.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.Optimization.Evolutionary.IntegrationTests\bin\Release\AwtSG.Domain.Optimization.Evolutionary.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.Optimization.Evolutionary.IntegrationTests\bin\Release\AwtSG.Domain.TechnicalPerformance.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.Optimization.Evolutionary.UnitTests\bin\Release\AwtSG.Domain.MetOceanData.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.Optimization.Evolutionary.UnitTests\bin\Release\AwtSG.Domain.Optimization.Evolutionary.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.Optimization.Evolutionary.UnitTests\bin\Release\AwtSG.Domain.TechnicalPerformance.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.Optimization.IntegrationTests\bin\Release\AwtSG.Domain.MetOceanData.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.Optimization.Mesh\bin\Release\AwtSG.Domain.NavigationUtilities.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.Simulation.IntegrationTests\bin\Release\AwtSG.Domain.MetOceanData.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.Simulation.UnitTests\bin\Release\AwtSG.Domain.NavigationUtilities.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.Simulation.UnitTests\bin\Release\AwtSG.Domain.Simulation.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.TechnicalPerformance.UnitTests\bin\Release\AwtSG.Domain.TechnicalPerformance.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Domain.Units.UnitTests\bin\Release\AwtSG.Domain.Units.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Dto.Mapping.UnitTests\bin\Release\AwtSG.Domain.NavigationUtilities.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Dto.Mapping.UnitTests\bin\Release\AwtSG.Domain.Simulation.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Dto.Mapping.UnitTests\bin\Release\AwtSG.Dto.Mapping.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Dto.Simulation.Mapping.UnitTests\bin\Release\AwtSG.Domain.NavigationUtilities.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Dto.Simulation.Mapping.UnitTests\bin\Release\AwtSG.Domain.Simulation.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Dto.Simulation.Mapping.UnitTests\bin\Release\AwtSG.Dto.Simulation.Mapping.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Dto.Simulation.UnitTests\bin\Release\AwtSG.Dto.Simulation.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Dto.UnitTests\bin\Release\AwtSG.Dto.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Infrastructure.UnitTests\bin\Release\AwtSG.Infrastructure.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.Numerics.UnitTests\bin\Release\AwtSG.Numerics.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.VoyageSimulationValidation\bin\Release\AwtSG.Numerics.UnitTests.dll" "C:\Tfs_Agent5\_work\1\s\src\AwtSG.WindowsService.VoyageSimulation.UnitTests\bin\Release\AwtSG.WindowsService.VoyageSimulation.UnitTests.dll"  /Settings:"C:\Tfs_Agent5\_work\1\s\src\all.runsettings" /EnableCodeCoverage /InIsolation /logger:trx /TestAdapterPath:"C:\Tfs_Agent5\_work\1\s\src\packages"
2017-04-20T16:51:13.4090709Z Microsoft (R) Test Execution Command Line Tool Version 14.0.25420.1
2017-04-20T16:51:13.4090709Z Copyright (c) Microsoft Corporation.  All rights reserved.
2017-04-20T16:51:15.7373080Z Starting test execution, please wait...
2017-04-20T16:51:19.4718867Z Warning: Diagnostic data adapter message: Could not find diagnostic data adapter 'Code Coverage'.  Make sure diagnostic data adapter is installed and try again.
2017-04-20T16:51:33.2378718Z Information: [xUnit.net 00:00:01.2430136]   Discovering:


2017-04-20T17:17:09.1501081Z Warning: System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain. This can happen if the test(s) started a thread but did not stop it. Make sure that all the threads started by the test(s) are stopped before completion.
2017-04-20T17:17:10.3845539Z Total tests: 17704. Passed: 17679. Failed: 0. Skipped: 25.
2017-04-20T17:17:10.3845539Z Test Run Successful.
2017-04-20T17:17:10.3845539Z Test execution time: 25.8603 Minutes
2017-04-20T17:17:28.5726606Z Results File: C:\Tfs_Agent5\_work\1\TestResults\tfsservice_US-SUN-TFSBUILD 2017-04-20 09_57_25.trx
2017-04-20T17:17:29.3539333Z Publishing Test Results...
2017-04-20T17:17:44.9950924Z Test results remaining: 17704
2017-04-20T17:17:47.0264093Z Test results remaining: 16704
2017-04-20T17:17:49.0421061Z Test results remaining: 15704
2017-04-20T17:17:53.1985047Z Test results remaining: 14704
2017-04-20T17:17:54.9329389Z Test results remaining: 13704
2017-04-20T17:17:56.5579944Z Test results remaining: 12704
2017-04-20T17:17:58.2299179Z Test results remaining: 11704
2017-04-20T17:17:59.9331076Z Test results remaining: 10704
2017-04-20T17:18:01.5894343Z Test results remaining: 9704
2017-04-20T17:18:03.0113618Z Test results remaining: 8704
2017-04-20T17:18:04.3395079Z Test results remaining: 7704
2017-04-20T17:18:05.6052151Z Test results remaining: 6704
2017-04-20T17:18:06.8083476Z Test results remaining: 5704
2017-04-20T17:18:08.5896555Z Test results remaining: 4704
2017-04-20T17:18:09.9178475Z Test results remaining: 3704
2017-04-20T17:18:11.2304148Z Test results remaining: 2704
2017-04-20T17:18:12.5429604Z Test results remaining: 1704
2017-04-20T17:18:13.8867197Z Test results remaining: 704
2017-04-20T17:18:25.5277535Z Published Test Run :

Note the warning that VS2015 Update 1 must be installed. This is what the About dialog looks like on the build agent (Update 3 is installed):

Solution

Parallel run is available in VS2015 Update 1 and later, make sure that you are using the right VS version to run the test in parallel.