I am currently reading into building a custom UI using Wix#, but since I have more experience with (WPF) MVVM than with WinForms i was wondering if it was possible to use that instead.
Is it possible?
You can actually use WPF UI with Wix#. See example in wix# repo.
If you want to use WinForms mandatory, you can also try to utilize WinForms bindings to write UI-related stuff in a WPF-style. See here about WinForms bindings. But in my opinion, WinForms bindings is not so good and I personally prefer not to use them in simple UI.