Search code examples
azurekubernetesgatewayazure-app-gateway-for-containers

TLS Termination in K8S Gateway API (azure implementation) Resource?

we are in the process of setting up a Azure Application Gateway for Containers, which is an implementation of the k8s Gateway API.

I have been able to set up a working POC with TLS termination in the Gateway and multi-site hosting. However, the problem I have is that I want to deploy these resources using our custom helm chart used for app deployments.

The way I have been able to set it up so far, is to add multiple TLS certificates in the gateway resource:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    alb.networking.azure.io/alb-name: alb-test-spoke-2
    alb.networking.azure.io/alb-namespace:system
  name: alb-gw-tls
  namespace:system
spec:
  gatewayClassName: azure-alb-external
  listeners:
  - allowedRoutes:
      namespaces:
        from: All
    hostname: first-api.dev
    name: https-listener-first-api
    port: 443
    protocol: HTTPS
    tls:
      certificateRefs:
      - kind: Secret
        name: first-api-tls-certificate
      mode: Terminate
  - allowedRoutes:
      namespaces:
        from: All
    hostname: second-api.dev
    name: https-listener-second-api
    port: 443
    protocol: HTTPS
    tls:
      certificateRefs:
      - kind: Secret
        name: second-api-tls-certificate
      mode: Terminate

This works, but it is not afaik feasible to make this work with a helm chart, as this approach requires me to add a new route/tls cert ref to the gateway for every new app manually, which I don't want to do.

I could create a new gateway for every application, but for some reason Azure limits the number of gateways (frontends) connected to AGC to just 5, which makes this approach very awkward.

I have tried looking into terminating the TLS in the HttpRoute Resource but that does not seem like a supported feature.

Does anyone else have a suggestion on how I can create these resources as standalone resources (files), thus making it easy to deploy using our helm-chart?

Solution

  • As discussed, Azure Application Gateway for Containers has a limit of five frontends per gateway, which currently cannot be increased. To work around this limitation, you can use a wildcard certificate, allowing a single Gateway resource to handle TLS termination for multiple subdomains. With this setup, you only need to manage HTTPRoute resources for each new app, eliminating frequent updates to the main Gateway configuration.

    Install the Gateway API CRDs on your cluster:

    kubectl apply -k "github.com/kubernetes-sigs/gateway-api/config/crd/experimental?ref=v0.5.0"

    If you have a wildcard certificate, create a secret for it, or if you’re using individual certificates for each hostname, create secrets for each certificate.

    enter image description here enter image description here

    Update your Gateway yaml with TLS termination using the wildcard or individual certificates.

    apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    alb.networking.azure.io/alb-name: alb-test-spoke-2
    alb.networking.azure.io/alb-namespace: system
  name: alb-gw-tls
  namespace: system
spec:
  gatewayClassName: azure-alb-external
  listeners:
    - name: https-listener-first-api
      port: 443
      protocol: HTTPS
      hostname: "first-api.dev"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: first-api-tls-certificate
            namespace: system
      allowedRoutes:
        namespaces:
          from: All
    - name: https-listener-second-api
      port: 443
      protocol: HTTPS
      hostname: "second-api.dev"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: second-api-tls-certificate
            namespace: system
      allowedRoutes:
        namespaces:
          from: All

    enter image description here

    Create a Helm Chart for the Gateway to simplify adding multiple listeners dynamically:

    enter image description here

    Edit the below values in the values.yaml file:

    listeners:
  - hostname: "first-api.dev"
    tlsSecretName: "first-api-tls-certificate"
  - hostname: "second-api.dev"
    tlsSecretName: "second-api-tls-certificate"

    And your templates/gateway.yaml:

    apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    alb.networking.azure.io/alb-name: alb-test-spoke-2
    alb.networking.azure.io/alb-namespace: system
  name: alb-gw-tls
  namespace: system
spec:
  gatewayClassName: azure-alb-external
  listeners:
    {{- range .Values.listeners }}
    - name: https-listener-{{ .hostname | replace "." "-" }}
      port: 443
      protocol: HTTPS
      hostname: {{ .hostname }}
      tls:
        mode: Terminate
        certificateRefs:
          - name: {{ .tlsSecretName }}
            kind: Secret
            namespace: system
      allowedRoutes:
        namespaces:
          from: All
    {{- end }}

    Deploy the Gateway Chart:

    helm install alb-gateway ./gateway-chart -n system

    This command will create the Gateway with listeners for first-api.dev and second-api.dev. If you need to add additional listeners in the future, simply update values.yaml.

    For each application, create a separate HTTPRoute resource that references the Gateway. This setup allows you to route traffic to new applications without modifying the Gateway configuration.

    apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: first-api-route
  namespace: app-namespace
spec:
  parentRefs:
    - name: alb-gw-tls
      namespace: system
  hostnames:
    - "first-api.dev"
  rules:
    - matches:
        - path:
            type: Prefix
            value: /
      backendRefs:
        - name: first-api-service
          port: 80

    enter image description here