Kubernetes service session affinity, how to stick session by service instead of endpoint

From my understanding, when setup kubernetes service with session affinity equal to "clientIP", the internal iptables writes rule to nat traffic by endpoint(service ip and port). So we have two related connection request to different port of the same service, it's very likely the second request will be redirect to another pod.

For example: Service S has port 1000 and 1001, 3 pods A,B,C are covered. What we might want is: 2 requests (R1 to port 1000, R2 to 1001) from client(IP:xx.xx.xx.xx) are sent to service S, the 2 request should be always redirected to pod A.

Kubernetes now treats S:1000 and S:1001 as two different endpoints(and the session affinity rule apply to them separately), so the 1st request will be sent to pod A and the 2nd request will be sent to pod B.

Is there anyway we could achieve the goal: stick session from same ip to same service to same pod?

Thanks in advance

Solution

I think you can use Nginx (like you correctly mentioned as internal LB) alongside with the Nginx Sticky sessions

The ingress controller replies the response with a Set-Cookie header to the first request. The value of the cookie will map to a specific pod replica. When the subsequent request come back again, the client browser will attach the cookie and the ingress controller is therefore able to route the traffic to the same pod replica.

The configuration is normally achieved with annotations.

Full list :

nginx.ingress.kubernetes.io/affinity
nginx.ingress.kubernetes.io/affinity-mode
nginx.ingress.kubernetes.io/session-cookie-name
nginx.ingress.kubernetes.io/session-cookie-path
nginx.ingress.kubernetes.io/session-cookie-samesite
nginx.ingress.kubernetes.io/session-cookie-conditional-samesite-none
nginx.ingress.kubernetes.io/session-cookie-max-age
nginx.ingress.kubernetes.io/session-cookie-expires
nginx.ingress.kubernetes.io/session-cookie-change-on-failure

Example: By setting the annotations in the ingress object, we can make sure the subsequent request will still be served by the same pod.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nginx-test
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "route"
    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"

spec:
  rules:
  - host: stickyingress.example.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /

To read: Sticky Sessions in Kubernetes

to check: kubernetes load balancer with sticky session always send traffic to one pod

Hope that helped