I want to verify a certificate that comes from a SAML assertion (issued by AZURE AD)
They can rollover the public key at any time. I am assuming it means that I cannot manage them manually in the trusted root store.
I am using X509Certificate2.Verify() which is returning False if the certificate is not in the Trusted root store. Should I use X509Certificate2.Build() and set it to ignore if it is trusted?
SAML2 uses certificates as a convenient way of representing encryption/signing keys with a standard file format. The contents of the certificate are not relevant because the certificates should be directly exchanged between the involved parties. So there is no need to verify the certificate's validity, in many deployments it won't even validate.
With an Idp that regularly does key rollover you should load and trust the metadata of the Idp instead of directly configuring the certificate.
If you are implementing all of this on your own I'd recommend that you spend some time looking for an existing implementation. Making a complete SAML2 implementation that supports loading metadata is non-trivial and takes a lot of time.