When .NET custom action projects are built using the WiX toolset, it appears to merge in all potentially needed code from direct project references while discarding code paths that will not be needed. Is this a correct interpretation of the behavior, and if so, what kind of shortcomings might there be (such as reflection issues)?
Can this system be somehow leveraged for more general usage outside of installer projects in place of ILMerge?
AFAIK makesfxca.exe (that's the tool your talking about, aren't you?) is not merging dll's in a pure technical sense. It rather takes its own native "stub"-dll named SfxCA.dll appends / packs other dependencies (e.g. WIX's own Microsoft.Deployment.WindowsInstaller as well as the managed dll containing the IL-code for the custom action) creating a native dll capable of "self extraction". On installing an msi this archive is temporarly extracted.
Thus it "resolves" the project references by copying the dll's passed in to makesfxca being able to omit the inclusion of just referenced but not locally copied .Net framework dll's. I'm not sure whether it really will discard anything.
ILMergeon the other side handles managed dll files, doing a real merge, rather than packing and later on extracting files.