SSL certificate generated by Firebase Hosting does not include connected domain

We've had this issue twice during the last week. For a Firebase Hosting project with two connected domains, one domain is not included in the certificate.

Trying to connect with a browser seems to return a 503 status code and Chrome shows net::ERR_CERT_COMMON_NAME_INVALID in the console. curl returns

(51) SSL: no alternative certificate subject name matches target host name '{host}'

(where {host} is the host name / connected domain)

To check the certificate directly, i.e. the SANs, I use the following command:

gnutls-cli --print-cert ${host} < /dev/null \
    | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' \
    | openssl x509 -noout -text \
    | grep DNS | tr , '\n' | tr -s " "

This returns a list of 100 certificates including the host name for the working domain, but only the default firebaseapp.com/*.firebaseapp.com entries for the failing domain.

Note: I am using gnutls-cli here, because it seems that openssl s_client -connect ${host}:443 does not include the host name in the request and always loads a certificate for firebaseapp.com/*.firebaseapp.com

I have already reached out to Firebase Support, but their last response (~16 hours ago) was "there are two different domains associated to the same project, but I need to confirm if this is supported". I am quite sure this is supported, given that during my analysis of the problem I have found more than 400 SANs for the same two host names next to the domains under our responsibility.

Any suggestions on how we can resolve this issue? I already tried deleting and re-adding the custom domain, but this did not change anything.

Switching hosting would technically not be too difficult, but our main issue is that the DNS is under control of a service provider of our customer and it's difficult for them to change anything already in production.

Solution

After some back and forth with Firebase Support, they found out that the root domain contained a CAA record that did not include letsencrypt.org.

The fix in our specific case was to include let's encrypt for the subdomain only. As an example, using the following settings

Domain: awesomesite.com
Connected Domain: firebaseapp.awesomesite.com

We can query the records using dig:

$ dig CAA awesomesite.com +noall +answer && dig CAA firebaseapp.awesomesite.com +noall +answer             

; <<>> (...) <<>> CAA awesomesite.com +noall +answer
;; global options: +cmd
awesomesite.com.        299 IN  CAA 0 iodef "mailto:[email protected]"
awesomesite.com.        299 IN  CAA 0 issue "digicert.com"

; <<>> (...) <<>> CAA firebaseapp.awesomesite.com +noall +answer
;; global options: +cmd
firebaseapp.awesomesite.com.    3599    IN  CAA 0 issue "letsencrypt.org"

As you can see, the domain firebaseapp.awesomesite.com has a CAA record with letsencrypt.org, while awesomesite.com was not touched.

Everything works fine now, just a short time after updating the records. We did not have to retrigger deployment on Firebase Hosting or delete/add the connected domain (things I had previously tried to resolved the problem).

Alternatives to correct the issue:

Delete CAA records: Remove the CAA record from the root domain (or intermediary domains).
Extend CAA records: Include letsencrypt.org in the list of domains in the CAA record of the root domain

Update, September 2020: According to an email I have received, Google is in the process of migrating the SSL certificate provider to a new Google-run CA because of Let's Encrypt's transition to ISRG's Root that may break backwards compatibility. According to that email, an additional CAA using pki.goog is required:

firebaseapp.awesomesite.com.    3599    IN  CAA 0 issue "pki.goog"
firebaseapp.awesomesite.com.    3599    IN  CAA 0 issue "letsencrypt.org"