When I browse my website from Chrome, it says that the certificate is invalid, and if I check the details, this is what I see:
Issued to:
Common Name (CN) test.x.example.com
Organization (O) cert-manager
Organizational Unit (OU) <Not Part Of Certificate>
Issued by:
Common Name (CN) cert-manager.local
Organization (O) cert-manager
Organizational Unit (OU) <Not Part Of Certificate>
I don't understand what is going wrong. From cert-manager's output it would seem everything is going well:
I1002 15:56:52.761583 1 start.go:76] cert-manager "level"=0 "msg"="starting controller" "git-commit"="95e8b7de" "version"="v0.9.1"
I1002 15:56:52.765337 1 controller.go:169] cert-manager/controller/build-context "level"=0 "msg"="configured acme dns01 nameservers" "nameservers"=["10.44.0.10:53"]
I1002 15:56:52.765777 1 controller.go:134] cert-manager/controller "level"=0 "msg"="starting leader election"
I1002 15:56:52.767133 1 leaderelection.go:235] attempting to acquire leader lease cert-manager/cert-manager-controller...
I1002 15:56:52.767946 1 metrics.go:203] cert-manager/metrics "level"=0 "msg"="listening for connections on" "address"="0.0.0.0:9402"
I1002 15:58:18.940473 1 leaderelection.go:245] successfully acquired lease cert-manager/cert-manager-controller
I1002 15:58:19.043002 1 controller.go:109] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="challenges"
I1002 15:58:19.043050 1 base_controller.go:132] cert-manager/controller/challenges "level"=0 "msg"="starting control loop"
I1002 15:58:19.043104 1 controller.go:91] cert-manager/controller "level"=0 "msg"="not starting controller as it's disabled" "controller"="certificates-experimental"
I1002 15:58:19.043174 1 controller.go:109] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="orders"
I1002 15:58:19.043200 1 base_controller.go:132] cert-manager/controller/orders "level"=0 "msg"="starting control loop"
I1002 15:58:19.043376 1 controller.go:109] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="certificates"
I1002 15:58:19.043410 1 base_controller.go:132] cert-manager/controller/certificates "level"=0 "msg"="starting control loop"
I1002 15:58:19.043646 1 controller.go:91] cert-manager/controller "level"=0 "msg"="not starting controller as it's disabled" "controller"="certificaterequests-issuer-ca"
I1002 15:58:19.044292 1 controller.go:109] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="clusterissuers"
I1002 15:58:19.044459 1 base_controller.go:132] cert-manager/controller/clusterissuers "level"=0 "msg"="starting control loop"
I1002 15:58:19.044617 1 controller.go:109] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="ingress-shim"
I1002 15:58:19.044742 1 base_controller.go:132] cert-manager/controller/ingress-shim "level"=0 "msg"="starting control loop"
I1002 15:58:19.044959 1 controller.go:109] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="issuers"
I1002 15:58:19.045110 1 base_controller.go:132] cert-manager/controller/issuers "level"=0 "msg"="starting control loop"
E1002 15:58:19.082958 1 base_controller.go:91] cert-manager/controller/certificates/handleOwnedResource "msg"="error getting order referenced by resource" "error"="certificate.certmanager.k8s.io \"api-certificate\" not found" "related_resource_kind"="Certificate" "related_resource_name"="api-certificate" "related_resource_namespace"="staging" "resource_kind"="Order" "resource_name"="api-certificate-3031097725" "resource_namespace"="staging"
I1002 15:58:19.143501 1 base_controller.go:187] cert-manager/controller/orders "level"=0 "msg"="syncing item" "key"="staging/api-certificate-3031097725"
I1002 15:58:19.143602 1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.143677 1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I1002 15:58:19.144011 1 sync.go:304] cert-manager/controller/orders "level"=0 "msg"="need to create challenges" "resource_kind"="Order" "resource_name"="api-certificate-3031097725" "resource_namespace"="staging" "number"=0
I1002 15:58:19.144043 1 logger.go:43] Calling GetOrder
I1002 15:58:19.144033 1 conditions.go:154] Setting lastTransitionTime for Certificate "cert-manager-webhook-webhook-tls" condition "Ready" to 2019-10-02 15:58:19.144027373 +0000 UTC m=+86.444394730
I1002 15:58:19.145112 1 conditions.go:154] Setting lastTransitionTime for Certificate "cert-manager-webhook-ca" condition "Ready" to 2019-10-02 15:58:19.145103359 +0000 UTC m=+86.445470721
I1002 15:58:19.145593 1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="staging/api-certificate"
I1002 15:58:19.147411 1 issue.go:169] cert-manager/controller/certificates/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="api-certificate-3031097725" "related_resource_namespace"="staging"
I1002 15:58:19.148059 1 base_controller.go:187] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.148099 1 base_controller.go:187] cert-manager/controller/ingress-shim "level"=0 "msg"="syncing item" "key"="staging/example-ingress"
I1002 15:58:19.148906 1 sync.go:71] cert-manager/controller/ingress-shim "level"=0 "msg"="not syncing ingress resource as it does not contain a \"certmanager.k8s.io/issuer\" or \"certmanager.k8s.io/cluster-issuer\" annotation" "resource_kind"="Ingress" "resource_name"="example-ingress" "resource_namespace"="staging"
I1002 15:58:19.148925 1 base_controller.go:193] cert-manager/controller/ingress-shim "level"=0 "msg"="finished processing work item" "key"="staging/example-ingress"
I1002 15:58:19.148133 1 base_controller.go:187] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-selfsign"
I1002 15:58:19.148963 1 conditions.go:91] Setting lastTransitionTime for Issuer "cert-manager-webhook-selfsign" condition "Ready" to 2019-10-02 15:58:19.148956891 +0000 UTC m=+86.449324275
I1002 15:58:19.149567 1 setup.go:73] cert-manager/controller/issuers/setup "level"=0 "msg"="signing CA verified" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-ca" "related_resource_namespace"="cert-manager" "resource_kind"="Issuer" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager"
I1002 15:58:19.149759 1 conditions.go:91] Setting lastTransitionTime for Issuer "cert-manager-webhook-ca" condition "Ready" to 2019-10-02 15:58:19.149752693 +0000 UTC m=+86.450120071
I1002 15:58:19.148155 1 base_controller.go:187] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="default/letsencrypt-staging"
I1002 15:58:19.150457 1 setup.go:160] cert-manager/controller/issuers "level"=0 "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="default" "resource_kind"="Issuer" "resource_name"="letsencrypt-staging" "resource_namespace"="default"
I1002 15:58:19.148177 1 base_controller.go:187] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="staging/letsencrypt-staging-issuer"
I1002 15:58:19.148630 1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="staging/api-certificate"
I1002 15:58:19.150669 1 base_controller.go:193] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="default/letsencrypt-staging"
I1002 15:58:19.151696 1 setup.go:160] cert-manager/controller/issuers "level"=0 "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging-secret-key" "related_resource_namespace"="staging" "resource_kind"="Issuer" "resource_name"="letsencrypt-staging-issuer" "resource_namespace"="staging"
I1002 15:58:19.151975 1 base_controller.go:193] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="staging/letsencrypt-staging-issuer"
I1002 15:58:19.153763 1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I1002 15:58:19.156512 1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.157047 1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I1002 15:58:19.157659 1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.158671 1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.158827 1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I1002 15:58:19.171562 1 base_controller.go:193] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.172759 1 base_controller.go:187] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.173387 1 setup.go:73] cert-manager/controller/issuers/setup "level"=0 "msg"="signing CA verified" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-ca" "related_resource_namespace"="cert-manager" "resource_kind"="Issuer" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager"
I1002 15:58:19.173465 1 base_controller.go:193] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.173562 1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I1002 15:58:19.174168 1 sync.go:329] cert-manager/controller/certificates/certificates "level"=0 "msg"="certificate scheduled for renewal" "duration_until_renewal"="6905h41m20.825882558s" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-webhook-tls" "related_resource_namespace"="cert-manager"
I1002 15:58:19.174487 1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I1002 15:58:19.175092 1 base_controller.go:193] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-selfsign"
I1002 15:58:19.175489 1 base_controller.go:187] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-selfsign"
I1002 15:58:19.175743 1 base_controller.go:193] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-selfsign"
I1002 15:58:19.175978 1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.176791 1 sync.go:329] cert-manager/controller/certificates/certificates "level"=0 "msg"="certificate scheduled for renewal" "duration_until_renewal"="41945h41m15.823245228s" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-ca" "related_resource_namespace"="cert-manager"
I1002 15:58:19.177118 1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.807942 1 base_controller.go:193] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="staging/api-certificate-3031097725"
Here is my configuration.
Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-ingress
annotations:
kubernetes.io/ingress.class: nginx
spec:
tls:
- hosts:
- test.x.example.com
secretName: letsencrypt-staging-certificate-secret
rules:
- host: test.x.example.com
http:
paths:
- path: /
backend:
serviceName: example-frontend
servicePort: 80
Issuer
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
name: letsencrypt-staging-issuer
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: [email protected]
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging-secret-key
# Enable the HTTP-01 challenge provider
solvers:
- http01: {}
Certificate
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: test-x-example-com
spec:
secretName: letsencrypt-staging-certificate-secret
issuerRef:
name: letsencrypt-staging-issuer
kind: Issuer
dnsNames:
- test.x.example.com
acme:
config:
- http01:
ingressClass: nginx
domains:
- test.x.example.com
Additional details: the secrets are in the staging namespace, like everything else except cert manager which is in the cert-manager namespace. The cluster is deployed on GKE.
EDIT: I'm wondering if it's possible that I hit the limits of the production environment in Let's Encrypt and got blocked. Is it possible to verify that somewhere?
I finally solved the issue mostly by editing the Certificate configuration. I also switched from an Issuer to a ClusterIssuer but that should not have any impact on this issue. I think the problem was ACME verification.
Here is my new ClusterIssuer:
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging-issuer
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: [email protected]
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging-secret-key
# Enable the HTTP-01 challenge provider
http01: {}
and, more importantly, the new Certificate:
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: test-x-example-com
spec:
secretName: letsencrypt-staging-certificate-secret
issuerRef:
name: letsencrypt-staging-issuer
kind: ClusterIssuer
dnsNames:
- test.x.example.com
acme:
config:
- http01:
ingressClass: nginx
domains:
- test.x.example.com