WiX [#filekey] is not being resolved and usable as a path in a managed CustomAction

Question

I am having a problem with using a deferred CustomAction where the parameter is passed as a file id reference in the style [#file.Product.exe.config]:

<Component Id="comp.Product.exe.config" 
           Guid="{966DA007-E3F0-4BFE-94ED-63A66F82CA58}" 
           Win64="$(var.Win64)">
    <File Id="file.Product.exe.config" 
          Name="Product.exe.config" 
          Source="$(var.SourcePath)\Product.exe.config" 
          KeyPath="yes" />
</Component>

Then I have a property called ConfigurationFile which is later on passed in to the deferred custom action like so:

<Property Id="ConfigurationFile" 
          Value="[#file.Product.exe.config]" />

I previously assigned the ConfigurationFile property Value="[APPLICATIONROOTDIRECTORY]\Product.exe.config" which worked just fine, but because we need to install the configuration to a different folder (the reason why this had to be done is irrelevant to this topic), I had to change it to resolve the location of the configuration file from the installer, using the [#file.Product.exe.config] syntax.

When the custom action receives the parameter, it is still in the [#file.Product.exe.config] format (that is, unresolved), and causing loads of problems. I was confused for quite a while because based on logging it seemed it should be working. It turns out that Session.Log() resolves this file reference, which resulted in the log "lying" to me about the actual content of the argument.

I have tried different approaches to "claning" this string, including session.Format() which results in InvalidHandleException, using record (as stated below) and other ways with no luck.

using (Record formatRec = new Record(0))
{
    formatRec.FormatString = p0;
    return formatRec.GetString(0) // Or formatRec.ToString();
}

There are several custom actions provided in WiX that handles this convention. However, I have yet to find out how they do it. No luck as of yet when diving into the source code for these custom actions.

Arguments to the custom action is retrieved from Session.CustomActionData.

Is there a nice way of handling this?

Answer 1

After a few grey hairs and quite a bit of foul language, I came up with a solution. It is based on @Zachary Youngs answer, however, some modifications were required.

Using a deferred action does not allow access to properties. Using the action as immediate type 51 action to set a property resulted in the property's [#filekey] reference remaining intact.

Using the following solution works:

<!-- Custom action to set update [ConfigurationFile] to the resolved filename -->
<CustomAction Id="SetResolvedConfigurationFile"
              BinaryKey="LocusCA"
              DllEntry='SetResolvedConfigurationFile'
              Return="check"
              Execute="immediate" />
<!-- Set CustomActionData for MergeConfiguratioFiles custom action -->
<CustomAction Id="SetMergeConfigurationFiles"
              Property="MergeConfigurationFiles"
              Execute="immediate"
              Value="[ConfigurationFile]" />

<!-- Merge settings from old configuration file to the one being installed -->
<CustomAction Id='MergeConfigurationFiles'
              BinaryKey="LocusCA"
              DllEntry='MergeConfigFiles'
              Return="check"
              Execute="commit" />

<InstallExecuteSequence>
    <Custom Action="SetResolvedConfigurationFile" Before="InstallFinalize">NOT (WIX_UPGRADE_DETECTED = "") AND NOT (ConfigurationFile = "")</Custom>
    <Custom Action="SetMergeConfigurationFiles" After="SetResolvedConfigurationFile">NOT (WIX_UPGRADE_DETECTED = "") AND NOT (ConfigurationFile = "")</Custom>
    <Custom Action="MergeConfigurationFiles" After="SetMergeConfigurationFiles">NOT (WIX_UPGRADE_DETECTED = "") AND NOT (ConfigurationFile = "")</Custom>
</InstallExecuteSequence>

The custom action SetResolvedConfigurationFile is as follows:

[CustomAction]
public static ActionResult SetResolvedConfigurationFile(Session session)
{
    // Set property ResolvedConfigurationFile
    try
    {
        session["ConfigurationFile"] = session.FormatRecord(new Record(1) { FormatString = session["ConfigurationFile"] });
    }
    catch (Exception ex)
    {
        session.Log("Unable to find configuration file for merge from property 'ConfigurationFile'): " + ex);
        return ActionResult.Failure;
    }
    return ActionResult.Success;
}

Using session.FormatRecord to resolve the current value of [ConfigurationFile], and write it back to the property, the existing setup of my custom action works.

Note: using the shortcut of changing my action MergeConfigurationFile to execute immediate does not work, as the files I need to merge do not yet exist. Using session.FormatRecord directly within MergeConfigurationFile results in a InvalidHandleException when trying to format when Execute="commit", and using Record.ToString() or Record.GetString() just returns the [#filekey] reference.