AKS Ingress-nginx takes no external traffic

Question

I have an AKS deployed in its own vnet (using Azure CNI instead of Kubenet). In the cluster there is an ingress-nginx deployed by Helm/ArgoCD with the annotation:

service.beta.kubernetes.io/azure-load-balancer-internal: true

The LoadBalancer looks good:

kubectl get svc -n ingress-nginx
NAME                     TYPE         CLUSTER-IP  EXTERNAL-IP
ingress.nginx-controller LoadBalancer 10.2.1.55   10.1.2.200

• I can curl any of those two IPs from within the cluster (any pod) and they work

• From a VM on the same vnet, i.e. outside the cluster, I can curl any pod directly

• I can curl any of the nginx daemonset pods directly on their internal IPs as well.

• BUT, I can't curl with the LoadBalancer IPs!

There is no traffic flowing at all, ends in a network timeout and no requests logs from nginx.

What am I missing? I am out of ideas.

Here's my very simple Helm input.

ingress-nginx:
  controller:
    hostPort:
      enabled: true
    kind: DaemonSet
    metrics:
      enabled: true
    publishService:
      enabled: false
    extraArgs:
      default-ssl-certificate: "ingress-nginx/ingress.local"
    service:
      annotations:
        service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    admissionWebhooks:
      enabled: false

Answer 1

Thanks to the blazingly fast and skillful K8s people in the r/kubernetes Reddit group this problem was solved.

When AKS is created there is a new Resource Group created for the cluster, even though you ask Azure to use a specific Resource Group. The new RG is named MC_rg_[cluster name]. Within this group there are among other things a health check created and it doesn't work. As long as the health probes aren't working, no traffic will be let through.

The Health Probes are found in the menu Insights -> Networking -> Load Balancer -> Kubernetes-internal

Just by replacing the http/https protocols to TCP in the probes, it started working. But a permanent solution is to set hard paths to the health URLs with the setting:

service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "healthz"

Here's an issue documenting the same thing: https://github.com/kubernetes/ingress-nginx/issues/8501

Hint: All logs from the cluster will end up in the new RG as well. You can write KQL queries directly to the logs for application logs, metrics, what not.