Search code examples
nginxsslkubernetes-helmazure-akscert-manager

Cert-Manager and Nginx under one umbrella helm chart in AKS fails to issue my certificates

My experience with Kubernetes is relatively small about a month since I have started learning and experimenting. I am migrating my setup from azure app services to AKS and I have encountered some issues with nginx ingress controller and cert-manager working together. Maybe it is the timing of the dns record change or my aproach of installing the packages that causes me the issue

My general approach is that I have a networking chart and an application chart meaning that I have to install one instance/release of the networking chart and I can install several instaces/release of my application chart(being staging,qa,production environments).

My networking chart looks like this:

Chart.yaml:

apiVersion: v2
name: networking
description: A Helm chart for Kubernetes
type: application
version: 0.0.1
appVersion: "1.1.0"
icon: <<I have iconurl here>>

dependencies:
  - name: nginx-ingress
    version: 0.14.0
    repository: https://helm.nginx.com/stable
    alias: nginx-ingress
  - name: cert-manager
    version: 1.8.2
    repository: https://charts.jetstack.io

values.yaml:

replicaCount: 1

cert-manager:
  installCRDs: true

nginx-ingress:
  controller:
    service:
      annotations: 
        "service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path": /healthz

I have no templates in this chart

The application chart has an issuer and ingress resource for the given environment as follows:

ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  {{ if eq .Values.environment "Release" }}
  namespace: release
  {{ else if eq .Values.environment "ReleaseQA" }}
  namespace: release-qa
  {{ else if eq .Values.environment "ReleaseProd" }}
  namespace: release-prod
  {{ else }}
  {{ required "value for .Values.environment is not as expected" .Values.environment }}
  {{ end }}
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    # ingress.kubernetes.io/ssl-redirect: "false" # I tried these various options
    cert-manager.io/issuer: letsencrypt-nginx
    # acme.cert-manager.io/http01-ingress-class: "nginx-cert-controller" # I tried these various options
spec:
  tls:
    {{ if eq .Values.environment "Release" }}
    - hosts:
        - core.staging.foo.com
      secretName: core-cert-nginx
    - hosts:
        - portal.staging.foo.com
      secretName: portal-cert-nginx
    - hosts:
#more tsl definitions here for other environments
#and the rules later in the file:
  ingressClassName: nginx
  rules:
    {{ if eq .Values.environment "Release" }}
    - host: core.staging.foo.com
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: core-service
                port: 
                  number: 80
    - host: portal.staging.foo.com
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: portal-service
                port: 
                  number: 80

issuer.yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-nginx
  {{ if eq .Values.environment "Release" }}
  namespace: release
  {{ else if eq .Values.environment "ReleaseQA" }}
  namespace: release-qa
  {{ else if eq .Values.environment "ReleaseProd" }}
  namespace: release-prod
  {{ else }}
  {{ required "value for .Values.environment is not as expected" .Values.environment }}
  {{ end }}
spec: 
  acme:
    email: <<my-valid-acme-email>>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-nginx-private-key
    solvers:
      # Use the HTTP-01 challenge provider
      - http01:
          ingress:
            class: nginx

All internal services and application pods are also created and healty.

So the question would be why is my http01 challenges keep failing? I also used in container terminal and descibed challenges with kubectl but I get the same result that the challenges created by cert-manager cannot be reached, in other words I get 302,404,502 codes or just routes to my application instead while switching between configurations.

I am also experiencing a weird behaviour that I cannot diagnose properly, that if the challenge does not fail at the first try it can pass and I am able to renew it with the cet-manager kubectl extention. Here I should mention that I am using external domain provider and I have to reroute traffic with the least down time so before the challenges can be completed they probably fail plenty of time before the domain address is set to the new IP.

I should probably mention the kubernetes environment details I am using:

Platform: Azure AKS

Kubernetes version: 1.23.5

nginx-ingress chart version: 0.14.0

cert-manager chart version: 1.8.2

applicaions running in the pods are:

  • 2*static react websites
  • 4*.NET Core backend apis and other type of services.

Please let me know in the comments/answers if you need addictional onformation to help.

Solution

  • I solved it by adding acme.cert-manager.io/http01-edit-in-place: "true" annotation on my ingress. For some reason the two different ingress resources made my ingress controller confused. But since then I moved on to dns01 challenges because they provide better versatility.