macos openssl ssl-certificate x509 secure-transport

macOS SecureTransport refuses to import a certificate accepted by OpenSSL

I'm trying to fix a failing test in the urllib3 open source Python project.

This test is called test_client_no_intermediate and tries to use a X.509 certificate called client_no_intermediate.pem. The goal is to fail with a "unknown CA" error. The test works as expected with OpenSSL: the import succeeds and the certificate is rejected.

However, on macOS with SecureTransport, the import fails with Import/Export format unsupported.. What's surprising is that it fails consistently on my machine with macOS 10.13.5 and Xcode 9.4.1 but only fails intermittently in continuous integration with macOS 10.12 and XCode 8.3.

Here is the certificate:

OpenSSL happily decodes it:

$ openssl x509 -noout -in client_no_intermediate.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            16:18:03:39:88:74:98:94:84:82:04:58:68:34:36:56:38:11:77:20:30:91:79:80:57:62:86:21:35:44:86:22:70:52:60:46:28:18
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=FI, ST=dummy, O=dummy, OU=dummy, CN=SnakeOil/emailAddress=dummy@test.local
        Validity
            Not Before: May 12 18:34:26 2017 GMT
            Not After : Dec 19 18:34:26 2021 GMT
        Subject: C=FI, ST=dummy, O=dummy, OU=dummy, CN=SnakeOilClient/emailAddress=dummy@test.local
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                    00:9a:76:46:cf:2d:73:df:03:56:cd:8e:02:fd:17:
                    af:ab:54:bb:37:b9:97:5b:5d:e9:87:7f:5a:0a:91:
                    3e:2a:f6:ea:ee:65:34:02:4c:31:6f:bd:61:ea:df:
                    6a:fa:1b:c2:95:ec:c0:12:6a:b8:92:cf:e6:c8:82:
                    9c:75:7a:e6:44:3a:6c:2e:71:df:94:2e:cd:07:31:
                    d7:ea:a2:b4:5a:dc:cb:94:b4:ab:88:cd:88:90:bc:
                    2b:2f:ef:cf:15:a0:d2:a3:b3:e3:4d:30:87:b1:3b:
                    aa:7a:4b:d1:37:e9:c7:b6:b6:bc:de:04:29:a9:62:
                    68:b4:5a:87:ce:d1:b4:b0:9f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Comment:
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier:
                6D:EF:6B:79:2C:36:0B:BD:89:B8:65:69:73:EF:CB:FC:ED:A1:56:3C
            X509v3 Authority Key Identifier:
                keyid:9E:5B:6C:9E:E8:76:9A:3B:6A:A8:35:A4:D9:13:52:C0:2E:E7:8C:79

    Signature Algorithm: sha256WithRSAEncryption
         0e:52:69:de:e3:0a:c6:04:b6:86:00:8a:f8:96:55:e2:ab:17:
         1a:e6:7f:ad:c1:e0:ab:9a:a5:ad:f8:34:8b:be:be:c3:06:7a:
         7e:cd:d6:41:ef:b3:36:05:db:26:74:6f:88:cd:15:49:c4:94:
         04:cf:08:b9:c9:72:a2:f1:56:2e:fb:34:67:4a:62:10:bb:59:
         ec:a3:f0:ac:35:5f:e5:2e:72:9c:91:e7:25:21:be:b2:3e:0d:
         ea:ff:ad:be:da:46:0d:bd:52:de:e7:8b:42:6e:fd:30:65:ef:
         f6:99:9b:69:80:66:50:9d:18:48:49:eb:49:c5:b2:f7:40:6f:
         4e:7a

But SecureTransport fails on this specific certificate:

$ security verify-cert -c cacert.pem
Cert Verify Result: CSSMERR_TP_NOT_TRUSTED
$ security verify-cert -c client_no_intermediate.pem
SecCertificateCreateFromData: Unknown format in import.

What is wrong in this certificate?

Solution

The only thing obviously "wrong" with the certificate is the serial number is 38 bytes long.

https://www.rfc-editor.org/rfc/rfc3280#section-4.1.2.2

Given the uniqueness requirements above, serial numbers can be expected to contain long integers. Certificate users MUST be able to handle serialNumber values up to 20 octets. Conformant CAs MUST NOT use serialNumber values longer than 20 octets.

While there is also guidance that non-conformant CAs exist, it's possible that Apple put in a limit lower than 38 bytes.

That wouldn't explain intermittency, but having a certificate which doesn't look "conformant" is always asking for unreliability.