Search code examples
google-cloud-platformgoogle-kubernetes-engineistiogke-networking

When I specify NodePort service as Ingress backend in GKE, I get two backends

-- Overview

I have a custom installation of istio on GKE (type=nodeport). The installation commands are as follows

istioctl install --set profile=default --set values.gateways.istio-ingressgateway.type=NodePort

I'm building Ingress and specifying the NodePort service as the backend. I found that another backend specified set by GCP as a default even though I specified NodePort. So, I couldn't connect to GCP LoadBalancer via TCP/IP. If I set the same port as the pod's readinessprobe for ingress and so on, the health checks look there too. Are there any way to solve this?

-- Detail

  • Detail view of an LB with two backends (the top backend service is the default configuration in GCP)

Detail view of LB

  • NodePort configuration
# This is a value that is automatically set by istio
$ k get svc istio-ingressgateway -n istio-system -o yaml
  ports:
  - name: status-port
    nodePort: 32476
    port: 15021
    protocol: TCP
    targetPort: 15021
  - name: http2
    nodePort: 32241
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    nodePort: 31739
    port: 443
    protocol: TCP
    targetPort: 8443
  - name: tcp-istiod
    nodePort: 32488
    port: 15012
    protocol: TCP
    targetPort: 15012
  - name: tls
    nodePort: 32741
    port: 15443
    protocol: TCP
    targetPort: 15443
  • Setting of readiness of istio-ingressgateway(pod)
$ k get po istio-ingressgateway-6f8bbbbd8c-qmkln -n istio-system -o yaml
:
    readinessProbe:
      failureThreshold: 30
      httpGet:
        path: /healthz/ready
        port: 15021
        scheme: HTTP
      initialDelaySeconds: 1
      periodSeconds: 2
      successThreshold: 1
      timeoutSeconds: 1
  • Setting of ingress
# 
spec:
  rules:
  - host: www.custom.com
    http:
      paths:
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 80
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 15021
  • Result of kubectl get svc
istio-system   istio-ingressgateway   NodePort    10.47.13.185   <none>        15021:31761/TCP,80:31561/TCP,443:31257/TCP,15012:31841/TCP,15443:32172/TCP   9h
Solution

  • I've divided this answer on parts:

    • Two backends when creating an Ingress resource with GKE.
    • Ingress definition in the question.
    • Integrating Cloud Armor with Istio.

    Two backends when creating an Ingress resource with GKE

    This is working as expected as Ingress created with GKE will have 2 backends:

    • The one specified in the YAML manifest.
    • The default backend: default-http-backend.

    As an example you can follow below steps:

    • $ kubectl create deployment nginx --image=nginx
    • $ kubectl expose deployment nginx --port=80 --type=NodePort
    • Create an Ingress resource for nginx but with a hello path (for example purposes)

    After that you should be seeing a similar setup:

    GCP LB

    The first backend service is using an Instance group to send the requests to the default-backend that are not matching the Ingress resource.

    The second backend service is using NEG (Network Endpoint Groups) to send the requests that are matching with the Ingress resource (in this example to an nginx Deployment).

    I've marked the red squares to "connect it" with the Kubernetes resources (look on ports):

    • $ kubectl get svc nginx
    NAME    TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
nginx   NodePort   10.20.6.229   <none>        80:32612/TCP   51m
    • $ kubectl get svc -n kube-system default-http-backend
    NAME                   TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
default-http-backend   NodePort   10.20.15.39   <none>        80:30603/TCP   121m

    Ingress definition in the question

    The part of the Ingress definition that was included in the question is/was created in a way that was referencing the same paths with 2 different backends:

        http:
      paths:
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 80
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 15021

    Above example:

    • Is having 2 different backends.
    • Is configured that 2 different backends share the same path.
    • The requests (as I've reproduced it) were coming to the "15021" backend which is only for health checking.

    To fix that, you will need to remove the 15021 backend and use backendConfig resource to configure health checking and security policy (more on that later).

    A side note!

    The YAML manifest is using and old and soon to be deprecated way to describe Ingress. Please refer to this documentation for more reference:

    Integrating Cloud Armor with Istio

    One of the ways that you can use Cloud Armor with Istio is the following:

    • Create a security policy.
    • Create a backendConfig as a prerequisite for Istio Service.
    • Install Istio with modification to its Service.
    • Create an Ingress resource to point to istio-ingressgateway.
    • Test with an example.

    Create a security policy

    For example purposes a security policy can be created that only blocks a single IP address. It can be done either by gcloud or Cloud Console (Web UI):

    Let's assume that the security policy named: deny-single was created that blocks a single IP address.

    Create a backendConfig as a prerequisite for Istio Service.

    You will need to create a backendConfig that will configure health checks and will enforce the security policy:

    apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: ingress-backendconfig
  namespace: istio-system
spec:
  healthCheck:
    requestPath: /healthz/ready
    port: 15021
    type: HTTP
  securityPolicy:
    name: deny-single # <-- IMPORTANT

    Install Istio with modification to its Service

    You will need to add following annotations to the Service of your istio-ingressgateway:

        cloud.google.com/backend-config: '{"default": "ingress-backendconfig"}'
    cloud.google.com/neg: '{"ingress":true}'

    This annotations will inform GCP on the security policy to apply as well on the health checks that are required for the traffic to be passed to the istio-ingressgateway.

    Create an Ingress resource to point to istio-ingressgateway

    A basic Ingress definition that will send the request from HTTP(S) Load Balancer to istio-ingressgateway can be following:

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: istio-ingress
  namespace: istio-system
spec:
  rules:
  - http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: istio-ingressgateway
            port:
              number: 80

    Test with an example

    To check if the setup is working correctly you can spawn the Bookinfo Application.

    Testing with 2 different IP addresses:

    • $ curl ifconfig.me
    217.AAA.BBB.CCC
    • $ curl 34.XXX.YYY.ZZZ/productpage
    <!doctype html><meta charset="utf-8"><meta name=viewport content="width=device-width, initial-scale=1"><title>403</title>403 Forbidden%
    • $ curl ifconfig.me
    94.EEE.FFF.GGG
    • $ curl 34.XXX.YYY.ZZZ/productpage
    <html>
  <head>
    <title>Simple Bookstore App</title>
<-- REDACTED -->

    A side note!

    The "closed port" that you've received could be related to the fact that istio-ingressgateway was configured to listen on specific path like /productpage and not /. (if the request was specifically targeted to port 80 instead of 15021)

    Additional resources: