Dynamic dependencies in batched targets / generating target chains from an item group

I'm trying to use target batching to factor out common code between similar target chains, while allowing them to be scheduled independently and have diverging dependencies. I've reduced my main issue to the following demo:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <ItemGroup>
        <Variants Include="VariantA;VariantB" TargetName="TargetFor%(Identity)" />
    </ItemGroup>

    <Target
        Name="BatchedTarget"
        BeforeTargets="%(Variants.TargetName)"
    >
        <Message Importance="High" Text="Common code running for %(Variants.Identity)" />
    </Target>

    <Target Name="TargetForVariantA">
        <Message Importance="High" Text="Inside TargetForVariantA" />
    </Target>

    <Target Name="TargetForVariantB">
        <Message Importance="High" Text="Inside TargetForVariantB" />
    </Target>
</Project>

When building the TargetForVariantA target, I would like BatchedTarget to execute only for the VariantA item. The BeforeTargets="%(Variants.TargetName)", which I wish would both batch BatchedTarget and setup the dependency properly, is not recognised and only produces a message visible in the .binlog file:

The target "%(Variants.TargetName)" listed in a BeforeTargets attribute at "C:\dev\nrm-evol\nrm\tests.proj (9,3)" does not exist in the project, and will be ignored.

What I've tried:

Using DependsOnTargets="BatchedTarget" on both TargetForVariant targets, of course, causes BatchedTarget to run for both variants.
Using BeforeTargets="@(Variants->'%(TargetName)')" on BatchedTarget manages to produce working dependencies, but understandably runs BatchedTarget for both variants as well.
Seeing that item transforms seems to work, I've tried the slightly cursed BeforeTargets="@(Variants->WithMetadataValue('Identity', '%(Variants.Identity)')->Metadata('TargetName'))", which turns out not to batch at all and expand to nothing (but no error message either this time).

Additional notes:

I would like these to remain data-driven, i.e. the Variants item group could be modified at evaluation time and include various metadata to drive the generated target chains.
I know running the MSBuild task on $(MSBuildProjectFullPath) with global properties to produce independent copies of the common target is a solution, but I'm afraid this will generate a huge amount of project instances (my real-world case is building 100~200 interdependent projects with a dozen different "variants"). The projects should also be able to override specific targets among the generated target chains, so I can't just isolate it in a separate .proj file and only MSBuild that.
The variants of BatchedTarget should be able to run at different points without waiting for one another in any way.

Is there any way to do this, or do I just have to accompany each addition to @(Variants) with a copy of BatchedTarget manually renamed for my new variant, and keep them all in sync?

Solution

Some general explanation about MSBuild is needed first.

MSBuild is not a procedural language; it's a declarative language.

MSBuild doesn't have subroutines. Targets are not subroutines.

MSBuild doesn't have branching. Because there is no branching, there are no flow control statements. That means there are no loop statements.

MSBuild has two datatypes: properties and items which are scalars and vectors, respectively.

There is no looping on items but there is 'batching' on items.

MSBuild has two phases when a project is 'run': Evaluation and Execution.

In the evaluation phase, properties and items that are not within a target are evaluated. The target build order is also determined in the evaluation phase.

In the execution phase, the targets are executed in the target build order. Properties and items within targets are evaluated when the target is executed.

Batching is not available in the evaluation phase. Batching only applies to the execution of targets and to the execution of tasks (which must be within targets).

Data driven targets from an Item

Given:

  <ItemGroup>
    <Variants Include="VariantA;VariantB" TargetName="TargetFor%(Identity)" />
  </ItemGroup>

it can be said that the Variants item has a VariantA item. But to be clearer, I will not overload 'item'. I will use 'Item Collection' and 'Item' and say that the Variants item collection has a VariantA item.

The target build order is created in the evaluation phase where batching can't be used. But that doesn't mean that an Item Collection can't be used to set a target order.

<!-- dynamic.proj -->
<Project>
    <PropertyGroup>
        <SelectValue Condition="'$(SelectValue)' == ''">VariantA</SelectValue>
    </PropertyGroup>

    <ItemGroup>
        <!-- Set up Variants -->
        <Variants Include="VariantA" TargetOrder="Apple;Cat" />
        <Variants Include="VariantB" TargetOrder="Boat;Dog" />
        <Variants Include="VariantB" TargetOrder="Cat" />
    </ItemGroup>

    <ItemGroup>
        <!-- Get Items from Variants that match $(SelectValue) -->
        <Selected Include="@(Variants->WithMetadataValue('Identity', $(SelectValue)))" />
    </ItemGroup>

    <PropertyGroup>
        <!-- Set MainDependsOn property from selected Variants items -->
        <MainDependsOn>@(Selected->'%(TargetOrder)')</MainDependsOn>
    </PropertyGroup>

    <Target Name="Main" DependsOnTargets="$(MainDependsOn)">
        <Message Text="Inside Target Main" />
    </Target>

    <Target Name="Apple" DependsOnTargets="GetFoo">
        <Message Text="Inside Target Apple" />
    </Target>

    <Target Name="Boat">
        <Message Text="Inside Target Boat" />
    </Target>

    <Target Name="Cat" DependsOnTargets="GetFoo" BeforeTargets="Dog">
        <Message Text="Inside Target Cat" />
    </Target>

    <Target Name="Dog">
        <Message Text="Inside Target Dog" />
    </Target>

    <Target Name="GetFoo">
        <Message Text="Inside Target GetFoo" />
    </Target>
</Project>

Assume the above MSBuild code is saved in a file named dynamic.proj.

The command msbuild dynamic.proj (or msbuild dynamic.proj -p:SelectValue=VariantA) will produce the following output:

GetFoo:
  Inside Target GetFoo
Apple:
  Inside Target Apple
Cat:
  Inside Target Cat
Main:
  Inside Target Main

The command msbuild dynamic.proj -p:SelectValue=VariantB will produce the following output:

Boat:
  Inside Target Boat
GetFoo:
  Inside Target GetFoo
Cat:
  Inside Target Cat
Dog:
  Inside Target Dog
Main:
  Inside Target Main

Some things to note from the example:

The item function WithMetadataValue is used to perform the 'lookup' in the item collection.
Item collections allow duplicate values for Identity. WithMetadataValue returns both items for VariantB. For VariantB the MainDependsOn property has a value of Boat;Dog;Cat.
Apple and Cat both depend on GetFoo. The GetFoo target will be run once for the project.
DependsOnTargets will add targets to the target build order. BeforeTargets and AfterTargets only affect the order of targets. The Cat target has BeforeTargets="Dog". For VariantA, the Cat target is run and the Dog target is not run. For VariantB, the Cat and Dog targets are both run and the order is changed from Boat;Dog;Cat to Boat;Cat;Dog.

Code Re-use

Is a data driven approach (like the above) good for code re-use? Generally, no.

By design, projects are isolated and independent from each other. There is no shared global data. A very large item collection that is defined in a shared file will be created new in every project that uses the shared file. If there is only one item in the item collection that is of need for a specific project, then the time and space used by creating and accessing the item collection can be considered wasteful.

MSBuild is declarative and it is XML. The project itself is the data.

As an example, the following file named shared.Targets replaces the use of an item collection with just directly setting the MainDependsOn property based on the SelectValue property.

<!-- shared.Targets -->
<Project>
    <PropertyGroup>
        <SelectValue Condition="'$(SelectValue)' == ''">VariantA</SelectValue>
    </PropertyGroup>

    <PropertyGroup>
        <!-- Set MainDependsOn property from $(SelectValue) property -->
        <MainDependsOn Condition="'$(SelectValue)' == 'VariantA'">Apple;Cat</MainDependsOn>
        <MainDependsOn Condition="'$(SelectValue)' == 'VariantB'">Boat;Dog;Cat</MainDependsOn>
    </PropertyGroup>

    <Target Name="Apple" DependsOnTargets="GetFoo">
        <Message Text="Inside Target Apple" />
    </Target>

    <Target Name="Boat">
        <Message Text="Inside Target Boat" />
    </Target>

    <Target Name="Cat" DependsOnTargets="GetFoo" BeforeTargets="Dog">
        <Message Text="Inside Target Cat" />
    </Target>

    <Target Name="Dog">
        <Message Text="Inside Target Dog" />
    </Target>

    <Target Name="GetFoo">
        <Message Text="Inside Target GetFoo" />
    </Target>
</Project>

A project, in this example projA, declares the build variant it wants and imports shared.Targets.

<!-- projA.proj -->
<Project>
    <PropertyGroup>
        <SelectValue>VariantA</SelectValue>
    </PropertyGroup>

    <Target Name="Main" DependsOnTargets="$(MainDependsOn)">
        <Message Text="Inside Target Main" />
    </Target>

    <Import Project="shared.Targets" />
</Project>

The command msbuild projA.proj produces the output:

GetFoo:
  Inside Target GetFoo
Apple:
  Inside Target Apple
Cat:
  Inside Target Cat
Main:
  Inside Target Main

A more common approach is not to try to define the target build order. Instead define the target's dependencies on each other and optionally define properties to enable or disable certain steps.

The shared file might look like:

<!-- shared.targets -->
<Project>
    <Target Name="Apple" DependsOnTargets="GetFoo">
        <Message Text="Inside Target Apple" />
    </Target>

    <Target Name="Boat">
        <Message Text="Inside Target Boat" />
    </Target>

    <Target Name="Cat" DependsOnTargets="Apple;GetFoo" BeforeTargets="Dog">
        <Message Text="Inside Target Cat" />
    </Target>

    <Target Name="Dog" DependsOnTargets="Boat;Cat">
        <Message Text="Inside Target Dog" />
    </Target>

    <Target Name="GetFoo">
        <Message Text="Inside Target GetFoo" />
    </Target>
</Project>

And the project file might look like:

<!-- projA.proj -->
<Project>
    <Target Name="Main" DependsOnTargets="Cat">
        <Message Text="Inside Target Main" />
    </Target>

    <Import Project="shared.targets" />
</Project>

Note that the files have become smaller and the output is the same.

Targets can be redefined

Let's say that projA.proj needs a different GetFoo.

<!-- projA.proj -->
<Project>
    <Target Name="Main" DependsOnTargets="Cat">
        <Message Text="Inside Target Main" />
    </Target>

    <Import Project="shared.targets" />

    <Target Name="GetFoo">
        <Message Text="Inside Target GetFoo as defined by projA" />
    </Target>
</Project>

After the Import, projA.proj redefines GetFoo.

The output of msbuild projA.proj will be:

GetFoo:
  Inside Target GetFoo as defined by projA
Apple:
  Inside Target Apple
Cat:
  Inside Target Cat
Main:
  Inside Target Main

If the alternative version of GetFoo is needed by more than one project, it can be placed in its own file that is imported.

SDK type projects are an example of code re-use

With .Net the standard build steps are all shared imported code and an SDK-style project may consist only of a PropertyGroup.