google-cloud-platform google-cloud-load-balancer mtls gcp-load-balancer gcp-certificate-manager

Cannot get mTLS working on a Google Cloud Load Balancer using PrivateCA

I am trying to get mTLS working on a global HTTPS load balancer in Google Cloud, but the client connections are just being rejected with the error "client_cert_validation_failed".

I followed the exact instructions here and here but it appears the trust config supplied just isn't working.

The trust config looks like this:

trustStores:
- trustAnchors:
  - pemCertificate: |
      -----BEGIN CERTIFICATE-----
      MIICWzCCAgKgAwIBAgIUANxN5VlXyQgGZp4uZJi7iXnIivEwCgYIKoZIzj0EAwIw
      NjERMA8GA1UEChMIVmlnaWxhbnQxITAfBgNVBAMTGGxvbmRvbi5hcGlzLnZpcHJv
      Lm9ubGluZTAeFw0yMzA4MjExMDA1NTZaFw0yMzExMTkxMDA1NTVaMDYxETAPBgNV
      BAoTCFZpZ2lsYW50MSEwHwYDVQQDExhsb25kb24uYXBpcy52aXByby5vbmxpbmUw
      WTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAT/H53qlrvXeBXtrPi8qnigM/YXKKKg
      tfssUk1Vxz+LoVor3DT4duqwyTm94naykqlmV25IzU5BAknff07TGVv5o4HtMIHq
      MA8GA1UdDwEB/wQFAwMH3oAwOwYDVR0lBDQwMgYIKwYBBQUHAwEGCCsGAQUFBwMC
      BggrBgEFBQcDAwYIKwYBBQUHAwQGCCsGAQUFBwMIMBIGA1UdEwEB/wQIMAYBAf8C
      AQUwHQYDVR0OBBYEFHo4y3dPvnRlI622Ytm5x/8WKZ1yMB8GA1UdIwQYMBaAFHo4
      y3dPvnRlI622Ytm5x/8WKZ1yMEYGA1UdEQQ/MD2CH2FnZW50cy5sb25kb24uYXBp
      cy52aXByby5vbmxpbmWCGm1jZC5hZ2VudHMuY2Eudmlwcm8ub25saW5lMAoGCCqG
      SM49BAMCA0cAMEQCIBmn2170sahTzA0iBYuULNeywX+r8fX8JucglsMQNBT8AiAP
      vJwqN9I7bmTgItfpjozMIExZiSeiTn45TfEdNLNsTQ==
      -----END CERTIFICATE-----

NOTE: PEM is definitely correct.

I have a TLS policy which uses this:

gcloud beta network-security server-tls-policies describe london-ips-tls-policy --location=global

createTime: '2023-08-22T15:59:51.030272533Z'
mtlsPolicy:
  clientValidationMode: REJECT_INVALID
  clientValidationTrustConfig: projects/MY_PROJECT_NUM/locations/global/trustConfigs/london-ips-trust-config
name: projects/MY_PROJECT_ID/locations/global/serverTlsPolicies/london-ips-tls-policy
updateTime: '2023-08-22T16:28:04.303764856Z'

I then have a target-https-proxies which references this:

# ...
serverTlsPolicy: //networksecurity.googleapis.com/projects/MY_PROJECT_ID/locations/global/serverTlsPolicies/london-ips-tls-policy

However whenever I download one of the approved certificates like this one:

Using the go code:

// load key/cert pair
    clientCertForTls, err := tls.LoadX509KeyPair(os.ExpandEnv(*clientCertPath), os.ExpandEnv(*privateKeyPath))
    if err != nil {
        log.Fatalf("unable to load x509 pair, %v", err)
    }

    // http client with client certs
    mTLSClient := &http.Client{
        Transport: &http.Transport{
            TLSClientConfig: &tls.Config{
                ClientAuth:   tls.RequestClientCert,
                Certificates: []tls.Certificate{clientCertForTls},
            },
        },
    }

    // make request
    resp, err := mTLSClient.Get("https://OUR_URL.com/")
    if err != nil {
        log.Fatalf("unable to mtls ips, %v", err)
    }
    bodyDat, _ := io.ReadAll(resp.Body)
    log.Printf("body: %s", bodyDat)
    for k := range resp.Header {
        log.Printf("header[%s]: %s", k, resp.Header.Get(k))
    }
    if resp.StatusCode != http.StatusOK {
        log.Fatalf("failed to mtls ips, %s: %s", resp.Status)
    } else {
        log.Printf("ALL GOOD! :)")
    }

However the connection is always rejected by the server.

Using the logging query:

jsonPayload.@type="type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"

I can see the error "client_cert_validation_failed" in the rejection entry:

{
httpRequest: {
latency: "0s"
remoteIp: "REDACTED"
}
insertId: "REDACTED"
jsonPayload: {
@type: "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
backendTargetProjectNumber: "projects/MY_PROJECT_NUM"
remoteIp: "REDACTED"
statusDetails: "client_cert_validation_failed"
}
logName: "projects/MY_PROJECT_ID/logs/requests"
receiveTimestamp: "2023-08-22T17:00:55.441451401Z"
resource: {
labels: {
backend_service_name: ""
forwarding_rule_name: "lb-frontend-acme-protected"
project_id: "MY_PROJECT_ID"
target_proxy_name: ""
url_map_name: ""
zone: "global"
}
type: "http_load_balancer"
}
severity: "INFO"
timestamp: "2023-08-22T17:00:54.413317Z"
}

Why does this not work please?

If I change the clientValidationMode to ALLOW_INVALID_OR_MISSING_CLIENT_CERT it allows traffic through, so for some reason it's rejecting perfectly good client certificates which were issued by the Root CA (which is enabled).

UPDATE: I've also tried with cURL, but the connection is still reset, the same as the Go code: curl --key ~/.vipro-key.pem --cert ~/Downloads/client-cert.crt -v https://OUR_PROTECTED_ENDPOINT/

Solution

I believe the concern here is the leaf certificate. The leaf certificate cannot contain the timeStamping in the x509v3 Extended Key Usage which falls under the Certificate Requirement

You can check this by running this command “openssl x509 -in cert.pem -text”