Since I migrated to WiX I only can run custom actions from binaries that are inserted in the Binary table
<Binary Id="SetupActions.CA.dll"
src="..\SetupActions\bin\Release\SetupActions.CA.dll" />
But Visual Studio Setup Project used to use the installed binaries as the container of custom actions.
Is there any way to use the old way in WiX?
You mean that you want to run a custom action that references a function within a dll that is installed with the package? In this case use the custom action type 17. Or in WiX:
<CustomAction Id="myCAfromInstalledDLL" FileKey="IdOfFile.dll" ExeCommand="EntryPointInDll" />