... and if yes, why can't I use SelfRegCost="0"?
So this might get a little philosophical ;)
The basic problem is: I have to register a ShellExtension DLL during a setup. When using plain old VS2008 deployment, I had a custom action which ran a regsvr32 on my dll. Then in my WiX setup I did the same, even with an unregister (/u) on uninstall. This works fine.
Until the other day I had a problem with some third-party DLLs loosing their registration after uninstalling a Visual Studio Setup. I found the installation had overwritten the default location of the DLLs. I discovered the possibility to change the Register Property from vsdrfCOMSelfReg to vsdrfDoNotRegister, and now my Setup did not overwrite the default location any more.
This made me thinking that I can use the same Property to register my ShellExtension DLL. So lets use dark on the VS Setup and see what the translation of the vsdrfCOMSelfReg is. And I found the only difference is the attribute SelfRegCost="0" in the File tag.
But according to this article ( How do you register a Win32 COM DLL file in WiX 3?) using SelfRegCost is baaad, lots of things go wrong. Possibly.
Now my reasoning is: either
a) dark.exe is not a realiable way to extract information from an MSI or
b) the use of SelfRegCost="0" is not as bad as everybody pretends, since Visual Studio does it eversince?
Visual Studio is exposing functionality; that doesn't mean it's a good idea to use it. Self-registration is bad and should be avoided by explicitly authoring COM registration using WiX elements like Class and ProgId.