I am not sure if I should ask this here or in Security Stackexchange perhaps.
In any event, I was recently working on RSA signatures using a TPM and came across an issue where I switched the padding scheme from RSASSA-PKCS1-v1_5 to RSASSA-PSS. I think this should not make a difference but I noticed one example in TSS.MSR (.NET TPM library) does not function anymore. I started an issue about it at https://github.com/microsoft/TSS.MSR/issues/109.
But I would like to check, if someone can share an opion, if there is a need to do or be mindful something obvious other than changing the padding scheme?
I think not, and this is implied also in parameters such as .NET RSA library, e.g. https://learn.microsoft.com/en-us/dotnet/api/system.security.cryptography.rsasignaturepadding and how one can use it like
using(var rsaKey = RSA.Create(keySizeInBits: 2048))
{
byte[] message = /* Random data. */
var sig = rsaKey.SignData(message, HashAlgorithmName.SHA256, RSASignaturePadding.Pss);
}
I see from questions in https://security.stackexchange.com/questions/183179/what-is-rsa-oaep-rsa-pss-in-simple-terms, https://crypto.stackexchange.com/questions/77881/are-rsa-pss-parameters-standardized and elsewhere how the padding actually happens is more complicated. But assuming it is an implementation detail of the library and it seem to be that signature checks do not match or work, one conclusion is that one might need to go check various internal parameters in this TSS.NET library, such as padding. So, hence I'd like to ensure myself this conclusion is correct enough and there isn't perhaps something very obvious. As for an illustration: do not use SHA-256 or put salt size to be exactly nn explicitly (OK, this is probably an implementation detail one shouldn't care about usually).
Addendum:
This is written after accepting the excellent notes by Maarten Bodewes.
Switching the hashing from SHA-256 to SHA-1 did not remove the failure on verifying signature in the linked example. Though, as expected, the "nameSize", or digest, changed to 20 bytes. So if there were some "lingering defaults" not properly handled in the example or the library somewhere, this alone (maybe a partial solution) was not the reason for the failure of switching to RSASSA-PSS padding scheme.
The quest continues. :)
Actually, there are some configuration options for the padding, and those do need to be established in advance when using the function.
Strangely enough, these "options" are not described for the PSS function in the standard. However, they are listed for the PSS padding mechanism:
Options: Hash hash function (hLen denotes the length in octets of the hash function output) MGF mask generation function sLen intended length in octets of the salt
However, this still doesn't configure PSS fully, as the MGF itself is an algorithm, and that algorithm has parameters as well.
So it is probably better to look at the ASN.1 structure of the parameters to get a good idea:
RSASSA-PSS-params ::= SEQUENCE { hashAlgorithm [0] HashAlgorithm DEFAULT sha1, maskGenAlgorithm [1] MaskGenAlgorithm DEFAULT mgf1SHA1, saltLength [2] INTEGER DEFAULT 20, trailerField [3] TrailerField DEFAULT trailerFieldBC }
There is fortunately only one Mask Generation Function: MGF1. There is only one trailer field specified as well: trailerFieldBC, so this is mainly specified for future changes.
As you can see there are two hash functions in there, one for the message / input data itself and one for the Mask Generation function (MGF). So the hash in turn is a configuration parameter for MGF1. The standard indicates that it is probably best to use the same hash function for both the message hash and the MGF1 hash. However, it may be that algorithms use the default instead, which is SHA-1 as you can see in the structure. AFAIK, Java always uses SHA-1 for the MGF1 but .NET uses the same hash as the message hash.
Similarly, the salt length is usually set to the hLen, the output size of the hash. It defaults to 20, but that's because that's the output size of SHA-1. The value of zero is also an option, but I think it is generally left to 20.
So yes, unfortunately you do need to check that the same parameters are used. A good library should clearly specify what is used or at least have the option to configure them explicitly. However, if that's the requirement for a good library, then many bad libraries exist. Moreover, you should definitely not try to gain this kind of information from the signature itself or guess multiple algorithms during verification, as that would lower the security to the least secure algorithm possible.
Finally, when you have found a secure and compatible PSS scheme I would recommend you to clearly specify in the code which set of parameters is used. This can be done using explicit parameters (even if the defaults are used for a particular library). You could also indicate the parameters in a comment if the former is not possible. It would not hurt to document your scheme separately and point to it in the comments.