IP Access restriction using Istio not working

I am trying to implement IP-based access restriction using Istio where I would like Istio to block all requests apart from Certain IPs. I got to a point where It's blocking the requests but it's not allowing the IPs I add in the YAML Manifest. I suspect the IP that it is seeing is different that the original Client IP as I have a couple of Load Balancers in front of my Istio.

I read from some documents that X-Forwarder-For is what we should be using to get the actual IP and I have used this in some other implementations but unclear how to get Istio to pick the actual X-Forwarder-Host. Can we create some custom headers and make Istio read from there? Please help

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ingress-policy
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  action: ALLOW
  rules:
  - from:
    - source:
        remoteIpBlocks: ["98.XXX.XXX.66"]

I enabled Debug Logs in my Envoy and below is what is getting recorded.

[2023-05-13T03:22:05.387Z] "GET /cluster/XXXX/XXXX/hari5 HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "98.XXX.XXX.66:52063,10.240.208.62" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" "37ccfaa6-7deb-414f-9f5c-72926c997994" "XXX-XX-XXX.XXX.com" "-" outbound|80||XXX.XXX-XXX.svc.cluster.local - 10.240.208.23:8080 10.240.208.62:40755 - -

Solution

This is what we had to use for restricting GET-access based on IP for one of our apps.

The x-forwarded-for header is just a comma-delimited string where first entry is the client IP-address, the remaining IP-addresses are from gateway, proxy etc.

For match with single IP-address: E.g. "123.234.345.567,*" (note the comma before the wild-card character is important when last segment is less than 3 digits)
For match with all addresses in last segment: E.g. "123.234.345.*"
For match with all addresses in 2 last segments: E.g. "123.234.*"

You may also add multiple IP-patterns under values.

I hope this will bring you further.

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: our-authorization-policy
  namespace: our-namespace
spec:
  selector:
    matchLabels:
      app: our-app-name
  action: ALLOW
  rules:
    - to:
        - operation:
            methods: ["GET"]
      when:
        - key: request.headers[x-forwarded-for]
          values:
            - "xxx.xxx.xxx.xxx,*" # note the trailing '*'