Avoid to bring MSI in compatibility mode "Previous version of Windows"

Question

I have an installer package for a 32-bit Application (built with MakeMsi, originally for Windows XP, and simplicisticly maintained since then), that fails registering a COM server on modern (64-bit) Windows systems (7, 8, 10). This is what I see when trying to install my MSI normally:

Application Error

Exception EOleSysError in module xyz at 000F0B01. Error accessing the OLE registry.

If I bring the MSI in compatibility mode Previous version of Windows, the COM server registers successfully. Since "it's working" somehow, I didn't invest much time in exploring the reasons so far. But finally, I'm exhausted to remember our customers (and sometimes also me) again and again of this precondition, so I wish to fix this issue.

The registration (and de-registration) is done via CustomActions, as I see looking into it using Orca:

"[INSTALLDIR.MYAPP]\placeholder.exe" -regserver
"[INSTALLDIR.MYAPP]\placeholder.exe" -unregserver

For each of those entries, Type is 1122 and Source is INSTALLDIR.MYAPP.

I could imagine that the COM server is started with insufficient privileges in the installation procedure, but aren't installers run automatically with administrator rights? I mean, when I (as a standard user) start the installer by double-clicking it, it shows the UAC prompt before the actual installation takes place. Why are the COM servers not run with elevated rights for their registration and de-registration? It's confusing...

How should I change my MSI to make Windows installer process it successfully?

Answer 1

After learning how to register COM servers the right way, I still was interested to understand, why my MSI package worked before. In other words: what is the crucial change after Windows XP?
...I meanwhile also checked that the MSI works when I run it as an administrator...

As I figured out reading the WiX toolset documentation, there is an attribute Impersonate for CustomAction that controls if the CA is executed with elevated privileges:

This attribute specifies whether the Windows Installer, which executes as LocalSystem, should impersonate the user context of the installing user when executing this custom action. Typically the value should be 'yes', except when the custom action needs elevated privileges to apply changes to the machine.

It refers to the msidbCustomActionTypeNoImpersonateflag in the Type field of the CustomAction Table. The value 1122 I observed in my MSI is decompiled into this:

• 1024 (flag): msidbCustomActionTypeInScript, or, deferred execution, runs in installation script
• 64 (flag): msidbCustomActionTypeContinue, or, ignore exit code and continue
• 34: Custom Action Type 34, or, launch executable via commandline

What I used to "fix" the problem the fast way is, enable the msidbCustomActionTypeNoImpersonate flag (2048) in the CA type (in WiX this would be Impersonate="no").

Translated into my MakeMsi script, I had to use the System attribute in the Type of the ExeCa command as to make self registration work as before:

Type="Deferred Sync AnyRc System"

I'm fully aware that this is only a workaround, since running a COM server for (un-)register is dangerous. And so the solution mentioned in the second section of PhilDWs answer should be preferred: manage COM related static information via registry entries in the MSI. But sometimes you need a fast solution, and sometimes there is no other option, see Euro Micellis comment.