This has been troubling me for some time.
Usually when you are searching in your trust store for the issuer certificate of another certificate in order to validate the signature, the first algorithm that comes to mind is to index your trust store by hashing the subject line of the certs so you can just hash the issuer and find its cert quickly. Then dealing with collisions by doing a memcmp() of the ASN.1 of the issuer of one, with the ASN.1 subject of the other; before checking the actual signature.
But, that is OK if both certificates to check have been issued by the same entity, but what happen if the issuer certificate is used to create another certificate using another piece of software that in the process changes one of the fields on the issuer from -let's say- IA5string (0x16) to Printable (0x13), Is it a correct behavior to fail the validation?
I mean, the OID and value will not change, just the ASN.1 encoding, but the result will be that the certificate won't be found (wrong hash) and will not validate.
Is this behavior correct?
The certificate is theoretically valid, as it has been signed with the correct key, so it does validate: (i.e. you first calculate the hash of 'tbsCertificate', use the public key of the issuer to obtain the hash from 'signatureValue', compare them and indeed verify that they match).
Nowhere I know of -although I have not searched exhaustively- there is an obligation for the issuer to use the exact same encoding on the issuer name than the one used on the subject name of the issuer cert. Actually, RFC-5280 says this in section 4.1.2.4 (Issuer):
When CAs have previously issued certificates with issuer fields with
attributes encoded using TeletexString, BMPString, or
UniversalString, then the CA MAY continue to use these encodings of
the DirectoryString to preserve backward compatibility.
Note that the word 'MAY' is in capitals, highlighting a free choice.
So I think that this freedom is why OpenSSL came up with the one-line thingy, to compare the Names after they have been decoded and therefore avoid encoding issues when searching for them.
But even if it is technically correct to do it, IMHO its not correct to modify the issuerName even if it is still correct after decoding, because it introduces a lot of possibilities that could potentially be used to find hash collisions and avoiding this should be made part of the validation. Which is why I am asking the question.
I am not asking what you think about this, I am asking someone more experienced to show me right or wrong.
Is it a correct behavior to fail in this situation?
DNs are compared using the rules specified in RFC 5280 Section 7.1:
7.1. Internationalized Names in Distinguished Names
Representation of internationalized names in distinguished names is
covered in Sections 4.1.2.4, Issuer Name, and 4.1.2.6, Subject Name.
Standard naming attributes, such as common name, employ the
DirectoryString type, which supports internationalized names through
a variety of language encodings. Conforming implementations MUST
support UTF8String and PrintableString. RFC 3280 required only
binary comparison of attribute values encoded in UTF8String, however,
this specification requires a more comprehensive handling of
comparison. Implementations may encounter certificates and CRLs with
names encoded using TeletexString, BMPString, or UniversalString, but
support for these is OPTIONAL.
The RFC then goes on to explain the comparison algorithm in detail, but from the above we can see that although strings must be decoded prior to comparison (i.e. it is not a binary comparison), support for deprecated encodings is OPTIONAL. Therefore, it is a valid behaviour for validation to fail in this scenario - some implementations may handle this, and others may not.