Adding session affinity via BackendConfig results in service outage (502)

Question

After adding session affinity via IPs to my service it resulted in 503s of my page I not yet understand why that happened.

The service itself did not throw any errors, but on the load balancer (LB) logs I see that the LB could not connect to the service anymore.

I am quite sure the outage was a result of adding the backend config, because the moment I removed the annotation the page recovered.

It would be really great if you could help me to find out why that happened and how I prevent that going forward, as I still want enable the session affinity.

---

Service annotations:

apiVersion: v1
kind: Service
metadata:
  annotations:
    beta.cloud.google.com/backend-config: '{"default": "SimonsBackendConf"}'
...

Backend config:

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: SimonsBackendConf
spec:
  sessionAffinity:
    affinityType: "CLIENT_IP"

Log entry which leads me to thinking it might be the service not being available:

{
  "jsonPayload": {
    "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
    "statusDetails": "backend_connection_closed_before_data_sent_to_client"
  },
  "httpRequest": {
    "status": 502,
...
  },
...
}

---

Setup:

• gke
• L7 google managed load balancer

Answer 1

In order for session affinity to work, you need to be running a VPC-native cluster as session affinity requires network endpoint groups. You will also need to create an Ingress resource for your service as well.

Assuming you have a VPC-native cluster, you'll need to add an additional annotation to your service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/backend-config: '{"default": "SimonsBackendConf"}'
    cloud.google.com/neg: '{"ingress": true}'
...

Note that the backend-config annotation is now GA as well (not sure which GKE version you are on)