Search code examples
firefoxtraefikx509acme

Smallstep X509 Certificates cause SEC_ERROR_BAD_DER in Firefox

I am setting up a PKI for local test environment. It's based on SmallStep and Traefik in Docker and I am trying to use name constraints to just the domains I use.

Template for root CA:

{
    "subject": {{ toJson .Subject }},
    "issuer": {{ toJson .Subject }},
    "keyUsage": ["certSign", "crlSign"],
    "basicConstraints": {
        "isCA": true,
        "maxPathLen": 1
    },
    "nameConstraints": {
        "critical": true,
        "permittedDNSDomains": ["myhost.local","localhost","127.0.0.1"]
    }
}

Template for intermediate CA:

{
    "subject": {{ toJson .Subject }},
    "keyUsage": ["certSign", "crlSign"],
    "basicConstraints": {
        "isCA": true,
        "maxPathLen": 0
    },
    "nameConstraints": {
        "critical": true,
        "permittedDNSDomains": ["myhost.local","localhost","127.0.0.1"]
    }

}

The certificates work fine with chromium based browsers, but cause SEC_ERROR_BAD_DER - improperly formatted DER-encoded message in Firefox.

The cert in question is:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 155623662314632950121135484293254901102 (0x751406f2098b9cc3b77eda93001e6d6e)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=Test3 CA Intermediate CA
        Validity
            Not Before: Dec 17 00:44:05 2024 UTC
            Not After : Dec 18 00:45:05 2024 UTC
        Subject: CN=nginx.myhost.local
        Subject Public Key Info:
            Public Key Algorithm: RSA
                Public-Key: (4096 bit)
                Modulus:
                    94:40:41:89:e2:57:9c:8b:be:cd:24:5f:56:86:0d:
                    bc:ce:5f:db:12:59:0f:d8:09:5b:e4:3a:95:88:e8:
                    f8:f7:34:9c:3e:c7:e0:d3:ad:a8:7d:8f:a7:4c:58:
                    92:7f:be:fd:54:9c:22:40:f8:00:58:7a:8b:fc:b4:
                    39:1f:9e:90:00:f5:3a:fb:be:bc:8a:4e:74:f3:71:
                    3b:f5:2a:58:a6:77:ce:1b:3c:c2:03:f9:97:7f:a6:
                    23:75:af:c0:6d:db:4f:44:8a:54:d5:5e:eb:6b:41:
                    34:7b:4c:28:da:65:21:81:30:61:b2:1e:83:5a:8d:
                    68:96:35:ed:47:90:a1:d1:2b:2a:c5:97:65:16:26:
                    7d:89:a2:4e:fc:b7:fd:ee:c5:80:f3:b9:73:7a:a6:
                    f6:a8:91:cb:ed:07:25:11:e9:3f:e6:0e:24:72:77:
                    2e:4a:44:03:74:fd:48:c0:e1:71:ad:01:8a:1e:15:
                    e5:3a:fe:25:86:97:e2:e7:30:ef:a4:e9:8b:13:d9:
                    3c:83:56:bf:25:19:35:35:51:c8:ea:b8:5c:5e:46:
                    08:f2:f7:77:11:bb:37:78:c9:6d:89:28:2b:09:9c:
                    53:8c:bb:16:67:e7:3d:72:3c:da:23:3b:68:fa:93:
                    c6:bc:51:d9:84:f4:04:c3:4b:08:ea:01:fd:9b:7d:
                    8c:46:35:93:ca:90:b8:15:00:67:64:37:df:2e:f3:
                    a6:7d:45:4e:91:9b:7c:71:8f:ca:5d:f9:04:a8:7a:
                    96:8d:35:f5:c5:21:56:be:40:d8:50:de:f8:78:b0:
                    9f:c1:b7:c4:44:2b:fe:e9:d0:02:ec:ff:25:2b:19:
                    e0:e9:4e:d7:30:b8:3d:0d:a0:50:56:31:b7:21:0f:
                    85:7d:df:8f:78:73:6f:4c:9e:0a:04:00:5d:20:70:
                    25:ad:43:8e:03:27:2d:12:2f:db:5c:2d:2e:34:e0:
                    fd:7e:ea:7a:54:0e:a5:ad:16:c4:d5:5b:01:0c:0c:
                    06:32:35:6c:ed:83:0e:56:16:f9:38:89:8b:58:53:
                    db:af:c9:d4:80:42:95:cd:03:ce:26:04:d8:16:49:
                    25:46:37:6e:bf:ae:db:66:3a:b0:5a:1a:22:4a:27:
                    18:fd:1e:43:63:de:ac:51:eb:b9:07:55:4e:58:5f:
                    ea:93:4a:c2:2c:02:6a:b3:a5:73:30:55:a3:6c:57:
                    73:fb:c7:39:57:52:52:ea:57:e0:d9:48:73:bc:e5:
                    e6:54:39:4f:ea:5a:18:c7:47:8f:50:3d:76:35:7c:
                    9c:d5:88:13:77:63:64:78:96:09:9d:a1:b6:7d:6a:
                    f8:a1:c1:a5:6b:9e:7c:25:7f:93:42:69:b5:b1:0a:
                    70:19
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                Server Authentication, Client Authentication
            X509v3 Subject Key Identifier:
                4F:C7:1D:2B:D0:C6:25:3C:7C:B4:B5:9F:26:7F:8C:03:31:1B:3E:40
            X509v3 Authority Key Identifier:
                keyid:40:2B:E8:0B:E4:39:91:9A:E1:7A:F0:D6:F6:00:0E:A6:2D:57:A9:05
            X509v3 Subject Alternative Name:
                DNS:nginx.myhost.local
            X509v3 Step Provisioner:
                Type: ACME
                Name: acme
    Signature Algorithm: ECDSA-SHA256
         30:46:02:21:00:bb:64:1d:ca:4e:11:5d:40:b2:a3:95:5d:7c:
         33:f5:cb:d0:f2:41:30:e4:61:32:d3:7d:ac:70:c6:60:df:9f:
         2d:02:21:00:c4:71:ab:7c:9e:3e:6d:cf:0f:43:64:4b:b9:77:
         5b:ff:c1:f1:88:f1:82:9c:b3:df:be:78:a8:24:d9:0b:bd:3e

Does anyone know what could be the issue?

I also tried openssl verify, without name constraints it looks OK:

openssl verify -verbose -show_chain -CAfile root_ca.crt -untrusted intermediate_ca.crt ./nginx-nonc.myhost.download.crt
./nginx-nonc.myhost.download.crt: OK
Chain:
depth=0: CN = nginx.myhost.local (untrusted)
depth=1: O = Test3, CN = Test3 Intermediate CA (untrusted)
depth=2: O = Test3, CN = Test3 Root CA

With name constraints it looks OK as well:

openssl verify -verbose -show_chain -CAfile ../Desktop/Work/step-ca-test/step-ca-example/data/step-ca/certs/root_ca.crt -untrusted ../Desktop/Work/step-ca-test/step-ca-example/data/step-ca/certs/intermediate_ca.crt ./nginx.myhost.download.crt
./nginx.myhost.download.crt: OK
Chain:
depth=0: CN = nginx.myhost.local (untrusted)
depth=1: O = Test3 CA, CN = Test3 CA Intermediate CA (untrusted)
depth=2: CN = Test3 Root CA
Solution

  • Well, I figured out the problem. The issue was, of course, with the firefox not using system's root CA cert.

    TLDR..

    It was necessary to use Firefox > Settings > Privacy & Security > Certificates > View Certificates... > Authorities > Import... to add the root CA cert.

    Surprisingly, deleting/distrusting the added root cert did nothing and the website is still trusted... but that might be just a caching issue.

    TL version

    I tried five scenarios (OS MS W11):

    Scenario 1

    • This scenario used Traefik only which generated its own default cert and served it for my nginx server on domain nginx.myhost.local.
    • Firefox and chrome based browsers did not trust it as it was self-signed. It must have been approved in advanced options to proceed.

    Scenario 2

    • This scenario used Traefik with SmallStep CA over ACME protocol. SmallStep CA was configured to serve signed certs with root CA cert (through intermediate CA cert).
    • In this case, firefox and chrome based browsers again did not trust it based on unknown CA. It must have been approved in advanced options to proceed.

    Scenario 3

    • This scenario was the same as the scenario 2 with root CA cert added to system's root CA store.
    • All browsers were OK with this.

    Scenario 4

    • This scenario was the same as the scenario 2 with root CA cert having name constraints on ".myhost.local".
    • In this case, firefox and chrome based browsers again did not trust it based on unknown CA. It must have been approved in advanced options to proceed.

    Scenario 5

    • This scenario was the same as the scenario 3 with root CA cert having name constraints on ".myhost.local".
    • In this case, chrome based browsers were OK with this, but Firefox threw SEC_ERROR_BAD_DER - improperly formatted DER-encoded message
    • It was necessary to use Firefox > Settings > Privacy & Security > Certificates > View Certificates... > Authorities > Import... to add the root CA cert.

    BTW. The intermediate.tpl was not used for these scenarios as according to RFC 5280 name constraints suffices to have in one cert of the trust chain.