Kubernetes: How to resolve HTTP 503 response from backend?

Question

Problem

I am getting a 503 response when my frontend container tries to contact an endpoint of my backend container while running them both in Kubernetes. When running the containers on my machine locally, all seems fine.

Details

My Spring backend has an endpoint "/hello", which returns "Hello World!". I have configured the CORS as flexible as possible in hope of ruling this out. Relevant code:

@Configuration
public class WebConfig implements WebMvcConfigurer {
    @Override
    public void addCorsMappings(CorsRegistry registry) {
        registry.addMapping("/**")
                .allowedMethods("*");
    }

    @Bean
    public CommonsRequestLoggingFilter logFilter() {
        var filter = new CommonsRequestLoggingFilter();
        filter.setIncludeQueryString(true);
        filter.setIncludePayload(true);
        filter.setMaxPayloadLength(10_000);
        filter.setIncludeHeaders(false);
        filter.setBeforeMessagePrefix("Request: ");
        return filter;
    }
}

My Kubernetes config is as follows:

backend.yaml

# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: knote-backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: knote
  template:
    metadata:
      labels:
        app: knote
    spec:
      containers:
        - name: backend
          image: knote-backend
          ports:
            - containerPort: 8080
          imagePullPolicy: Never
          resources:
            limits:
              cpu: "2"
              memory: "512Mi"
---
# Service
apiVersion: v1
kind: Service
metadata:
  name: knote-backend
spec:
  selector:
    app: knote
    tier: backend
  ports:
    - port: 8080
      targetPort: 8080

frontend.yaml

# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: knote-frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: knote
      tier: frontend
  template:
    metadata:
      labels:
        app: knote
        tier: frontend
    spec:
      containers:
        - name: frontend
          image: "knote-frontend"
          ports:
            - containerPort: 3000
          imagePullPolicy: Never
          resources:
            limits:
              cpu: "2"
              memory: "512Mi"
---
# Service
apiVersion: v1
kind: Service
metadata:
  name: knote-frontend
spec:
  selector:
    app: knote
    tier: frontend
  ports:
    - port: 3000
      targetPort: 3000

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: knote-ingress
  annotations:
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "http://192.168.49.2"
  labels:
    name: knote-ingress
spec:
  rules:
    - http:
        paths:
          - pathType: Prefix
            path: "/api"
            backend:
              service:
                name: knote-backend
                port:
                  number: 8080
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: knote-frontend
                port:
                  number: 3000

Output of "kubectl describe ingress knote-ingress":

Name:             knote-ingress
Labels:           name=knote-ingress
Namespace:        default
Address:          192.168.49.2
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /api   knote-backend:8080 (<none>)
              /      knote-frontend:3000 (10.244.0.64:3000)
Annotations:  nginx.ingress.kubernetes.io/cors-allow-origin: http://192.168.49.2
              nginx.ingress.kubernetes.io/enable-cors: true
Events:
  Type    Reason  Age                  From                      Message
  ----    ------  ----                 ----                      -------
  Normal  Sync    31m (x3 over 4h32m)  nginx-ingress-controller  Scheduled for sync

Tried

So far, I have tried

• Checking whether the containers work locally: they do
• Adding/removing the CORS annotations in ingress.yaml: no effect
• Adding allowedOrigins of "knote-frontend" and "192.168.49.2": no effect
• curl within knote-backend to localhost which works, but curl within knote-frontend fails

EDIT: This is the request header when trying to reach the '/hello' endpoint:

GET /api/hello HTTP/1.1
Host: 192.168.49.2
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/114.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://192.168.49.2/
DNT: 1
Connection: keep-alive
Sec-GPC: 1
Pragma: no-cache
Cache-Control: no-cache

It tries to reach /api/hello, which the ingress should redirect to knote-backend:8080, which is where the Spring backend should listen and reply with "Hello World!". The response header is a typical:

HTTP/1.1 503 Service Temporarily Unavailable
Date: Sun, 25 Jun 2023 12:46:20 GMT
Content-Type: text/html
Content-Length: 190
Connection: keep-alive

Answer 1

Maybe remove tier: backend from the backend's Service. or add tier: backend to your backend's Deployment. Your service is looking for two selectors. But only one of them is found.