Search code examples
istioistio-gateway

Inbound Istio Ingress Gateway Metrics

I deployed a custom Istio Ingress Gateway deployment with the default IstioOperator telemetry config and no EnvoyFilter to modify the stats.

When I checked the Prometheus stats via:

kubectl exec -n web-ingress-gateway web-ingress-gateway-6966569988-876mp -c istio-proxy -- pilot-agent request GET /stats/prometheus | grep istio_requests_total

It returns a bunch of

istio_requests_total{response_code="200",reporter="source",source_workload="web-ingress-gateway",source_workload_namespace="web-ingress-gateway",source_principal="unknown",source_app="web-ingress-gateway",source_version="unknown",source_cluster="redacted",destination_workload="redacted",destination_workload_namespace="redacted",destination_principal="unknown",destination_app="unknown",destination_version="unknown",destination_service="redacted",destination_service_name="redacted",destination_service_namespace="redacted",destination_cluster="redacted",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="unknown",source_canonical_service="web-ingress-gateway",destination_canonical_service="redacted",source_canonical_revision="latest",destination_canonical_revision="latest"} 28
...

It seems that only reporter="source" labels are there, but no reporter="destionation" which is usually also present in the sidecar.

Is there a way to get the incoming requests metrics?

I followed this doc.

But it does not really break down to the detail that I need, as it only gives you the response_code_class only.

# TYPE envoy_http_outbound_0_0_0_0_9282_downstream_rq counter
envoy_http_outbound_0_0_0_0_9282_downstream_rq{response_code_class="1xx"} 0
envoy_http_outbound_0_0_0_0_9282_downstream_rq{response_code_class="2xx"} 1062279
envoy_http_outbound_0_0_0_0_9282_downstream_rq{response_code_class="3xx"} 130245
envoy_http_outbound_0_0_0_0_9282_downstream_rq{response_code_class="4xx"} 12532
envoy_http_outbound_0_0_0_0_9282_downstream_rq{response_code_class="5xx"} 578
Solution

  • According to documentation:

    Reporter: This identifies the reporter of the request. It is set to destination if report is from a server Istio proxy and source if report is from a client Istio proxy or a gateway.

    This means that requests with reporter="source" are sent from source pod (web-ingress-gateway in your case), and reported by prometheus as such in this pod.

    Requests with reporter="destination" are sent from another pod to a desitnation pod in your case (that would be web-ingress-gateway -> ), and are reported as such in the destination pod.

    If you issue the same command, but to the application pod (not ingress-gateway), you will see requests with reporter="destination".

    Using the Bookinfo application as an example, we can se the requests from productpage to details pods
    bookinfo application

    productpage is a source pod, and is reported as reporter="source"
    productpage

    details is a destination pod, and is reported as reporter="destination"
    details

    In case I misunderstood your question, and you want to see metrics of requests coming from outside, to your ingress gateway - this is currently not possible. Ingress only emits metrics with reporter as source [source]