Dozens of Q&A entries (all but one in stackoverflow!) that I found got close to this question, but didn't teach me what I needed. I have what should be a nearly simple installation: a Windows service and an associated tray icon application. They install fine with the standard VS Setup project. After the files are installed, I need to present a dialog to the user to set some parameters in the service's exe.config file. In that dialog, the user should be able to abort the installation. I've tried two approaches to the Custom Action process and ran into a wall with each when it comes to making the installation roll back.
Approach 1: An exe for the Custom Action, run at Commit time. This sort-of works. The application returns a non-zero exit code and the installation rollback occurs. What I don't like is that:
Approach 2: An Installer as the Custom Action, run at Install or Commit time. This is cleaner to me (only the one item listed in the Custom Actions), but getting the process to roll back is worse than Approach 1. It seems that I have to throw an exception in the overridden method (Install/Commit), which then gives me several error dialogs before the rollback occurs, and then the rollback doesn't always uninstall the service.
What is the cleanest way to make this work without going to WiX or similar options?
The simple answer is don't use VS Installer Classes. They have a number of fundamental design flaws. I suggest you look at Windows Installer XML's ( WiX ) Deployment Tools Foundation ( DTF ) instead. DTF is a far superior hosting model / interop library for your managed code and outputs DLL's that are compatible ( from an MSI perspective ) with C/C++ custom actions. They can be consumed as Custom Actions by any MSI authoring tool, not just WiX installers.