When publishing, a Test folder is created with an msixbundle file for installing the app on my computer to test the app. It also creates a cer file. As far as I remember, the certificate didn't use to change. Now, every time I publish it, I need to install the certificate again because it has changed.
Is this a change in Visual Studio and there's nothing I can do about it, or is it something that I accidentally changed in the settings and I can change back?
You can specify which certificate to use when signing the package by selecting a PFX file under the Packaging tab in the app manifest designer in Visual Studio.
If you click on the "Choose Certificate..." button, you'll also have the option to create a self-signed test certificate.
You can also do this using the New-SelfSignedCertificate PowerShell cmdlet as explained here.
In a previous version of Visual Studio, there was indeed a default PFX file included in the project template but it seems to have been removed.