GKE Ingress with Multiple Backend Services returns 404

Question

I'm trying to create a GKE Ingress that points to two different backend services based on path. I've seen a few posts explaining this is only possible with an nginx Ingress because gke ingress doesn't support rewrite-target. However, this Google documentation, GKE Ingresss - Multiple backend services, seems to imply otherwise. I've followed the steps in the docs but haven't had any success. Only the service that is available on the path prefix of / is returned. Any other path prefix, like /v2, returns a 404 Not found.

Details of my setup are below. Is there an obvious error here -- is the Google documentation incorrect and this is only possible using nginx ingress?

-- Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: app-static-ip
    networking.gke.io/managed-certificates: app-managed-cert
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80
      - path: /v2
        pathType: Prefix
        backend:
          service:
            name: api-2-service
            port:
              number: 8080

-- Service 1
apiVersion: v1
kind: Service
metadata:
  name: api-service
  labels:
    app: api
spec:
  type: NodePort
  selector:
    app: api
  ports:
  - port: 80
    targetPort: 5000

-- Service 2
apiVersion: v1
kind: Service
metadata:
  name: api-2-service
  labels:
    app: api-2
spec:
  type: NodePort
  selector:
    app: api-2
  ports:
  - port: 8080
    targetPort: 5000

Answer 1

GCP Ingress supports multiple paths. This is also well described in Setting up HTTP(S) Load Balancing with Ingress. For my test I've used both Hello-world v1 and v2.

There are 3 possible issues.

• Issue is with container ports opened. You can check it using netstat:

$ kk exec -ti first-55bb869fb8-76nvq -c container -- bin/sh
/ # netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 :::8080                 :::*                    LISTEN      1/hello-app

• Issue might be also caused by the Firewall configuration. Make sure you have proper settings. (In general, in the new cluster I didn't need to add anything but if you have more stuff and have specific Firewall configurations it might block).

• Misconfiguration between port, containerPort and targetPort.

Below my example:

1st deployment with

apiVersion: apps/v1
kind: Deployment
metadata:
  name: first
  labels:
    app: api
spec:
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: container
          image: gcr.io/google-samples/hello-app:1.0
          ports:
          - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: api-service
  labels:
    app: api
spec:
  type: NodePort
  selector:
    app: api
  ports:
  - port: 5000
    targetPort: 8080

2nd deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: second
  labels:
    app: api-2
spec:
  selector:
    matchLabels:
      app: api-2
  template:
    metadata:
      labels:
        app: api-2
    spec:
      containers:
        - name: container
          image: gcr.io/google-samples/hello-app:2.0
          ports:
          - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: api-2-service
  labels:
    app: api-2
spec:
  type: NodePort
  selector:
    app: api-2
  ports:
  - port: 6000
    targetPort: 8080

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 5000
      - path: /v2
        pathType: Prefix
        backend:
          service:
            name: api-2-service
            port:
              number: 6000

Outputs:

$ curl 35.190.XX.249
Hello, world!
Version: 1.0.0
Hostname: first-55bb869fb8-76nvq
$ curl 35.190.XX.249/v2
Hello, world!
Version: 2.0.0
Hostname: second-d7d87c6d8-zv9jr

Please keep in mind that you can also use Nginx Ingress on GKE by adding specific annotation.

kubernetes.io/ingress.class: "nginx" 

Main reason why people use nginx ingress on GKE is using rewrite annotation and possibility to use ClusterIP or NodePort as serviceType, where GCP ingress allows only NodePort serviceType.

Additional information you can find in GKE Ingress for HTTP(S) Load Balancing