Istio ingress gateway randomly returns 507 response

Question

Ingress gateway is retrying if the upstream returns 502. Most of the time it is working as expected. Sometimes gateway returns 507 "exceeded request buffer limit while retrying upstream" without retrying.

From the logs I can see this: app 502 -> istio-proxy sidecar 502 -> ingress gateway 507 -> client. Unable to find other errors in logs related to ingress-gateway.

Requests are 1-30Mb in size. Any ideas where to look for the issue?

VirtualService:

...
      retries:
        attempts: 4
        retryOn: 502,retriable-status-codes,connect-failure
        retryRemoteLocalities: true
...

EnvoyFilter:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: buffer-limit
spec:
  configPatches:
    - applyTo: LISTENER
      listenerMatch: 0.0.0.0_8080
      patch:
        operation: MERGE
        value:
          per_connection_buffer_limit_bytes: 100000000

Answer 1

Solution was to use correct Envoy filter:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  namespace: istio-system
  name: buffer-limit
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: LISTENER
      patch:
        operation: MERGE
        value:
          per_connection_buffer_limit_bytes: 100000000

The filter is working on v1.8.4

To check if the filter is applied:

istioctl proxy-config listeners <istio-ingressgateway-pod> -o json -n istio-system