503 service temporary unavailable ingress EKS

Question

getting a 503 error for the ingress, did the basic trouble shooting with labels and stuff looks good though. I see the pods are running and can be listed when ran with the service label.

the readiness probe has a warning but it did not fail

what else can be checked tor resolve this issue. any ideas appreciated

kubectl get service -n staging
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
app-staging   ClusterIP   172.20.174.146   <none>        8000/TCP   242d


kubectl describe service app-staging -n staging
Name:              app-staging
Namespace:         staging
Labels:            <none>
Annotations:       <none>
Selector:          app=app-staging
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                172.20.174.146
IPs:               172.20.174.146
Port:              app-staging  8000/TCP
TargetPort:        8000/TCP
Endpoints:         10.200.32.6:8000,10.200.64.2:8000
Session Affinity:  None
Events:            <none>


kubectl get pods -n staging -l app=app-staging                     
NAME                                     READY   STATUS    RESTARTS   AGE
app-staging-5677656dc8-djp8l   1/1     Running   0          4d7h
app-staging-5677656dc8-dln5v   1/1     Running   0          4d7h

this is the readiness probe

 kubectl describe pod app-staging-5677656dc8-djp8l -n staging|grep -i readiness
    Readiness:      http-get http://:8000/ delay=30s timeout=1s period=30s #success=1 #failure=6
  Warning  ProbeWarning  40s (x12469 over 4d7h)  kubelet  Readiness probe warning:

here is the manifest file for the pod, service and ingress

# This deployment is setup to use ECR for now, but should switch to  Artifactory in the future.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-staging
  namespace: staging
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app-staging
  template:
    metadata:
      labels:
        app: app-staging
    spec:
      containers:
        - name: app-staging
          image: "${DOCKER_REGISTRY}/:${IMAGE_TAG}"
          readinessProbe:
            failureThreshold: 6
            httpGet:
              path: /
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 1
          imagePullPolicy: Always
         # Setting AUTODYNATRACE_FORKABLE environment variable will cause an ominous looking error message similar to the one below:
         #
         #  `WARNING autodynatrace - init: Could not initialize the OneAgent SDK, AgentState: 1`
         #
         # This error message is expected when "forkable" mode is enabled. See the link below for more information:
         # https://github.com/Dynatrace/OneAgent-SDK-for-Python/blob/fa4dd209b6a21407abca09a6fb8da1b85755ab0a/src/oneagent/__init__.py#L205-L217
          command: ["/bin/sh"]
          args:
            - -c
            - >-
                /bin/sed -i -e "s/# 'autodynatrace.wrappers.django'/'autodynatrace.wrappers.django'/" /app//ON_/ON_/settings.py &&
                /usr/local/bin/python manage.py collectstatic --noinput &&
                AUTOWRAPT_BOOTSTRAP=autodynatrace AUTODYNATRACE_FORKABLE=True /usr/local/bin/gunicorn --workers 8 --preload --timeout 120 --config gunicorn.conf.py --bind 0.0.0.0:8000
          env:
            - name: AUTODYNATRACE_POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: AUTODYNATRACE_APPLICATION_ID
              value: Django ($(AUTODYNATRACE_POD_NAME):8000)
          ports:
            - containerPort: 8000
          volumeMounts:
          # mount config in both locations while we migrate to running container as non-root user.
          - name: secrets
            readOnly: true
            mountPath: /root/FHIREngine/conf
          - name: secrets
            readOnly: true
            mountPath: /home//FHIREngine/conf
      imagePullSecrets:
        - name: jfrogcred
      volumes:
       - name: secrets
         secret:
            secretName: config
            defaultMode: 420
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: app-staging
  namespace: staging
spec:
  ports:
    - name: app-staging
      port: 8000
      targetPort: 8000
  selector:
    app: app-staging
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-staging
  namespace: staging
  annotations:
    external-dns.alpha.kubernetes.io/hostname: staging.tv-pd.sh.io
    external-dns.alpha.kubernetes.io/type: internal
    kubernetes.io/ingress.class: nginx-internal
spec:
  rules:
    - host: staging.tv-pd.sh.io
      http:
        paths:
          - path: /
            backend:
              service:
                name: app
                port:
                  number: 8000
            pathType: ImplementationSpecific
            #pathType is now required for each specified path. Options are Prefix, Exact, and ImplementationSpecific. To match the undefined v1beta1 behavior, use ImplementationSpecific
---

Answer 1

I see that your service is named "app-staging"

apiVersion: v1
kind: Service
metadata:
  name: app-staging

But in the ingress the path mapping to service is incorrectly identifying the service name as "app"

spec:
  rules:
    - host: staging.tv-pd.sh.io
      http:
        paths:
          - path: /
            backend:
              service:
                name: app
                port:
                  number: 8000

Please change the backend service name in ingress to "app-staging" instead of "app".

Please accept the answer if this resolves your issue.