AKS with application gateway - always 502

Question

I followed ms and k8s tutorials and tried to configure ingress for 2 demo rest apis, with application gateway. I hope to have chat-service at /chat and session-service at /session, but requests always end up as 502 bad gateway.

I tested my cluster with MS ingress demo app, and it worked. Looked the same as mine, except just 1 service.

...
apiVersion: apps/v1
kind: Deployment
metadata:
  name: session-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: session-service
  template:
    metadata:
      labels:
        app: session-service
    spec:
      nodeSelector:
        app: scalable
      containers:
      - name: session-service-container
        imagePullPolicy: Always
        image: myacr.azurecr.io/scalable/session-service:latest
        ports:
        - containerPort: 8080
        env:
        - name: REDIS
          value: "scalable-redis.default.svc.cluster.local"
        resources:
          requests:
            cpu: 1m
            memory: 128Mi
          limits:
            cpu: 30m
            memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
  name: session-service
spec:
  ports:
  - port: 8080
    targetPort: 8080
    protocol: TCP
  type: ClusterIP
  selector:
    app: session-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chat-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: chat-service
  template:
    metadata:
      labels:
        app: chat-service
    spec:
      nodeSelector:
        app: scalable
      containers:
      - name: chat-service-container
        imagePullPolicy: Always
        image: myacr.azurecr.io/scalable/chat-service:latest
        ports:
        - containerPort: 8080
        env:
        - name: REDIS
          value: "scalable-redis.default.svc.cluster.local"
        resources:
          requests:
            cpu: 1m
            memory: 128Mi
          limits:
            cpu: 30m
            memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
  name: chat-service
spec:
  type: ClusterIP
  ports:
  - port: 8080
    targetPort: 8080
    protocol: TCP
  selector:
    app: chat-service
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: scalable-ingress
spec:
  ingressClassName: azure-application-gateway
  rules:
  - http:
      paths:
      - path: /chat
        pathType: Prefix
        backend:
          service:
            name: chat-service
            port: 
              number: 8080
      - path: /session
        pathType: Prefix
        backend:
          service:
            name: session-service
            port: 
              number: 8080

If i port-forward those pods to local and make requests, they are working fine. What am I missing? health probes come up as "unhealthy", does that affect the routes?

$ kubectl get pods,svc,ingress
NAME                                   READY   STATUS    RESTARTS   AGE
pod/chat-front-5647f964bf-kt86z        1/1     Running   0          12h
pod/chat-front-5647f964bf-vkxhg        1/1     Running   0          12h
pod/chat-service-77cc84cf45-5ns8q      1/1     Running   0          12h
pod/chat-service-77cc84cf45-lqf6g      1/1     Running   0          12h
pod/scalable-redis-9f69d9f96-v7d69     1/1     Running   0          12h
pod/session-service-5b66cddcbf-sfmbg   1/1     Running   0          12h
pod/session-service-5b66cddcbf-z9v4h   1/1     Running   0          12h

NAME                      TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE
service/chat-front-lb     LoadBalancer   10.0.146.62    20.250.xxx.yyy   80:31316/TCP   12h
service/chat-service      ClusterIP      10.0.166.152   <none>           8080/TCP       12h
service/kubernetes        ClusterIP      10.0.0.1       <none>           443/TCP        12h
service/scalable-redis    ClusterIP      10.0.90.162    <none>           6379/TCP       12h
service/session-service   ClusterIP      10.0.55.242    <none>           8080/TCP       12h

NAME                                         CLASS                       HOSTS   ADDRESS        PORTS   AGE
ingress.networking.k8s.io/scalable-ingress   azure-application-gateway   *       4.226.xxx.yyy   80      12h

Answer 1

Argh, it was about the probes. the automatic configuration was messed up.

[edit] I had two things going wrong here:

a) the application gateway probes must return "healthy" for traffic to be sent to that backend. I had custom health endpoints, but with that yaml the probes were configured to ping the wrong address (needed to specify "/chat/health" instead of just "/chat" since my root was not responding anything

b) my services has routes like "/login", but when they go through prefix paths, the service receives calls to "/session/login" and there is no endpoint. I needed rewrite rules to strip out the path prefixes. Something like "if server variable uri_path is /session/(.+) then set url path to /{var_uri_path_1}"

then everything worked. after that i moved to having subdomains for each api, so I dont need to do path rewrites