Zonal network endpoint group unhealthy even though that container application working properly

Question

I've created a Kubernetes cluster on Google Cloud and even though the application is running properly (which I've checked running requests inside the cluster) it seems that the NEG health check is not working properly. Any ideas on the cause?

I've tried to change the service from NodePort to LoadBalancer, different ways of adding annotations to the service. I was thinking that perhaps it might be related to the https requirement in the django side.

# [START kubernetes_deployment]
apiVersion: apps/v1
kind: Deployment
metadata:
  name: moner-app
  labels:
    app: moner-app
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: moner-app
  template:
    metadata:
      labels:
        app: moner-app
    spec:
      containers:
      - name: moner-core-container
        image: my-template
        imagePullPolicy: Always
        resources:
          requests:
            memory: "128Mi"
          limits:
            memory: "512Mi"
        startupProbe:
          httpGet:
            path: /ht/
            port: 5000
            httpHeaders:
              - name: "X-Forwarded-Proto"
                value: "https"
          failureThreshold: 30
          timeoutSeconds: 10
          periodSeconds: 10
          initialDelaySeconds: 90
        readinessProbe:
            initialDelaySeconds: 120
            httpGet:
              path: "/ht/"
              port: 5000
              httpHeaders:
              - name: "X-Forwarded-Proto"
                value: "https"
            periodSeconds: 10
            failureThreshold: 3
            timeoutSeconds: 10
        livenessProbe:
          initialDelaySeconds: 30
          failureThreshold: 3
          periodSeconds: 30
          timeoutSeconds: 10
          httpGet:
            path: "/ht/"
            port: 5000
            httpHeaders:
              - name: "X-Forwarded-Proto"
                value: "https"
        volumeMounts:
          - name: cloudstorage-credentials
            mountPath: /secrets/cloudstorage
            readOnly: true
        env:
            # [START_secrets]
            - name: THIS_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: GRACEFUL_TIMEOUT
              value: '120'
            - name: GUNICORN_HARD_TIMEOUT
              value: '90'
            - name: DJANGO_ALLOWED_HOSTS
              value: '*,$(THIS_POD_IP),0.0.0.0'
        ports:
        - containerPort: 5000
        args: ["/start"]
        
      # [START proxy_container]
      - image: gcr.io/cloudsql-docker/gce-proxy:1.16
        name: cloudsql-proxy
        command: ["/cloud_sql_proxy", "--dir=/cloudsql",
                  "-instances=moner-dev:us-east1:core-db=tcp:5432",
                  "-credential_file=/secrets/cloudsql/credentials.json"]
        resources:
          requests:
            memory: "64Mi"
          limits:
            memory: "128Mi"
        volumeMounts:
          - name: cloudsql-oauth-credentials
            mountPath: /secrets/cloudsql
            readOnly: true
          - name: ssl-certs
            mountPath: /etc/ssl/certs
          - name: cloudsql
            mountPath: /cloudsql
      # [END proxy_container]
      # [START volumes]
      volumes:
        - name: cloudsql-oauth-credentials
          secret:
            secretName: cloudsql-oauth-credentials
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
        - name: cloudsql
          emptyDir: {}
        - name: cloudstorage-credentials
          secret:
            secretName: cloudstorage-credentials
      # [END volumes]
# [END kubernetes_deployment]

---

# [START service]
apiVersion: v1
kind: Service
metadata:
  name: moner-svc
  annotations:
    cloud.google.com/neg: '{"ingress": true, "exposed_ports": {"5000":{}}}' # Creates an NEG after an Ingress is created
    cloud.google.com/backend-config: '{"default": "moner-backendconfig"}'
  labels:
    app: moner-svc
spec:
  type: NodePort
  ports:
  - name: moner-core-http
    port: 5000
    protocol: TCP
    targetPort: 5000
  selector:
    app: moner-app
# [END service]

---

# [START certificates_setup]
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: managed-cert
spec:
  domains:
    - domain.com
    - app.domain.com
# [END certificates_setup]

---

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: moner-backendconfig
spec:
  customRequestHeaders:
      headers:
      - "X-Forwarded-Proto:https"
  healthCheck:
    checkIntervalSec: 15
    port: 5000
    type: HTTP
    requestPath: /ht/


---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: managed-cert-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: moner-ssl
    networking.gke.io/managed-certificates: managed-cert
    kubernetes.io/ingress.class: "gce"
spec:
  defaultBackend:
    service:
      name: moner-svc
      port: 
        name: moner-core-http

Answer 1

I'm still not sure why, but i've managed to work when moved the service to port 80 and kept the health check on 5000.

Service config:

# [START service]
apiVersion: v1
kind: Service
metadata:
  name: moner-svc
  annotations:
    cloud.google.com/neg: '{"ingress": true, "exposed_ports": {"5000":{}}}' # Creates an NEG after an Ingress is created
    cloud.google.com/backend-config: '{"default": "moner-backendconfig"}'
  labels:
    app: moner-svc
spec:
  type: NodePort
  ports:
  - name: moner-core-http
    port: 80
    protocol: TCP
    targetPort: 5000
  selector:
    app: moner-app
# [END service]

Backend config:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: moner-backendconfig
spec:
  customRequestHeaders:
      headers:
      - "X-Forwarded-Proto:https"
  healthCheck:
    checkIntervalSec: 15
    port: 5000
    type: HTTP
    requestPath: /ht/