Route request based on ssl_client_verify in Nginx ingress controller

Question

Depending on the result of ssl_client_verify, I want to use a different upstream. So if $ssl_client_verify = SUCCESS route to yes-mtls-backend:80 K8s service, otherwise route to no-mtls-backend:80 K8s service. The following did not work:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dynamic-upstream
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-tls-secret: default/ca-cert
    nginx.ingress.kubernetes.io/auth-tls-verify-client: optional
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "3"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ($ssl_client_verify = SUCCESS) {
        set $proxy_host "default-yes-mtls-backend-80";
      }
spec:
  rules:
  - host: xyz.com
    http:
      paths:
      - backend:
          service:
            name: no-mtls-backend
            port:
              number: 80
        path: /
        pathType: Prefix

Note that auth-tls-verify-client is optional so that Nginx will not ask for a client certificate.

How to get this working? Is $proxy_host the wrong thing to set or this whole approach is flawed?

Answer 1

This worked:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dynamic-upstream
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-tls-secret: default/ca-cert
    nginx.ingress.kubernetes.io/auth-tls-verify-client: optional
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "3"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ($ssl_client_verify = SUCCESS) {
        set $service_name   "yes-mtls-backend";
        set $service_port   "80";
        set $proxy_upstream_name "default-yes-mtls-backend-80";
      }
spec:
  rules:
  - host: xyz.com
    http:
      paths:
      - backend:
          service:
            name: no-mtls-backend
            port:
              number: 80
        path: /
        pathType: Prefix
  - host: xyz.com
    http:
      paths:
      - backend:
          service:
            name: yes-mtls-backend
            port:
              number: 80
        path: /
        pathType: Prefix

In my version of Nginx, upstreams are dynamically loaded using Lua, so duplicating the rule will tell Lua to load yes-mtls-backend in the upstream block. That way, when overwriting $service_name, $service_port and $proxy_upstream_name there will be a backend to route traffic to.