Search code examples
kuberneteskubernetes-ingressnginx-ingressk3s

ingress nginx how to debug 502 page even though the ports in service and Ingress are correct?

i have a web application running inside cluster ip on worker node on port 5001,i'm also using k3s for cluster deployment, i checked the cluster connection it's running fine

enter image description here

the deployment has the container port set to 5001:

ports:
  - containerPort:5001

Here is the service file:

apiVersion: v1
kind: Service
metadata:
  labels:
    io.kompose.service: user-ms
  name: user-ms
spec:
  ports:
    - name: http
      port: 80
      targetPort: 5001
  selector:
    io.kompose.service: user-ms
status:
  loadBalancer: {}

and here is the ingress file:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
   name: user-ms-ingress
spec:
   rules:
   - http:
      paths:
      - path: / 
        pathType: Prefix
        backend:
           service:
             name: user-ms
             port: 
               number: 80

i'm getting 502 Bad Gateway error whenever i type in my worker or master ip address

expected: it should return the web application page

i looked online and most of them mention wrong port for service and ingress, but my ports are correct yes i triple check it:

try calling user-ms service on port 80 from another pod -> worked try calling cluster ip on worker node on port 5001 -> worked

the ports are running correct, why is the ingress returning 502?

here is the ingress describe:

enter image description here

and here is the describe of nginx ingress controller pod:

enter image description here enter image description here

the nginx ingress pod running normally:

enter image description here

here is the logs of the nginx ingress pod:

enter image description here

sorry for the images, but i'm using a streaming machine to access the terminal so i can't copy paste

How should i go with debugging this error?

Solution

  • ok i managed to figure out this, in the default setting of K3S it uses traefik as it default ingress, so that why my nginx ingress log doesn't show anything from 502 Bad GateWay

    I decided to tear down my cluster and set it up again, now with suggestion from this issue https://github.com/k3s-io/k3s/issues/1160#issuecomment-1058846505 to create cluster without traefik:

    curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik" sh -

    now when i call kubectl get pods --all-namespaces i no longer see traefik pod running, previously it had traefik pods runining.

    once i done all of it, run apply on ingress once again -> get 404 error, i checked in the nginx ingress pod logs now it's showing new error of missing Ingress class, i add the following to my ingress configuration file under metadata:

    metadata:
  name: user-ms-ingress
  annotitations:
    kubernetes.io/ingress.class: "nginx"

    now i once more go to the ip of the worker node -> 404 error gone but got 502 bad gateway error, i checked the logs get connection refused errors:

    enter image description here

    i figured out that i was setting a network policy for all of my micro services, i delete the network policy and remove it's setting from all my deployment files.

    Finally check once more and i can see that i can access my api and swagger page normally.

    TLDR:

    1. If you are using nginx ingress on K3S, remember to disable traefik first when created the cluster
    2. don't forget to set ingress class inside your ingress configuration
    3. don't setup network policy because nginx pod won't be able to call the other pods in that network