Search code examples
prometheusistioenvoyproxy

Customizing Istio metrics results in duplicated metrics


I want to add request.url_path and request.method information to the istio_request_total metric by using an EnvoyFilter.

Istio Version: 1.8.5

Using my EnvoyFilter duplicates istio_requests_total metric by adding envoy_request_path__$(path-seperated-with-dashes)__ prefix.

Below is the EnvoyFilter I use

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: customize-requests-total-metric
spec:
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_OUTBOUND
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: istio.stats
        proxy:
          proxyVersion: ^1\.8.*
      patch:
        operation: REPLACE
        value:
          name: istio.stats
          typed_config:
            "@type": type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
            value:
              config:
                configuration:
                  "@type": type.googleapis.com/google.protobuf.StringValue
                  value: |
                    {
                      "debug": "true",
                      "stat_prefix": "istio",
                      "metrics": [
                        {
                          "name": "requests_total",
                          "dimensions": {
                            "request_path": "request.url_path",
                            "request_method": "request.method"
                          }
                        }
                      ]
                    }
                root_id: stats_outbound
                vm_config:
                  code:
                    local:
                      inline_string: envoy.wasm.stats
                  runtime: envoy.wasm.runtime.null
                  vm_id: stats_outbound
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
              subFilter:
                name: istio.stats
        proxy:
          proxyVersion: ^1\.8.*
      patch:
        operation: REPLACE
        value:
          name: istio.stats
          typed_config:
            "@type": type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
            value:
              config:
                configuration:
                  "@type": type.googleapis.com/google.protobuf.StringValue
                  value: |
                    {
                      "debug": "true",
                      "stat_prefix": "istio",
                      "metrics": [
                        {
                          "name": "requests_total",
                          "dimensions": {
                            "request_path": "request.url_path",
                            "request_method": "request.method"
                          }
                        }
                      ]
                    }
                root_id: stats_inbound
                vm_config:
                  code:
                    local:
                      inline_string: envoy.wasm.stats
                  runtime: envoy.wasm.runtime.null
                  vm_id: stats_inbound

This duplicates istio_requests_total metric by adding envoy_request_path__$(path-seperated-with-dashes)__ prefix. Below is an example of it.

# TYPE envoy_request_path____catalog_lint___istio_requests_total counter
envoy_request_path____catalog_lint___istio_requests_total{response_code="200",reporter="destination",source_workload="my-service",source_workload_namespace="my-namespace",source_principal="unknown",source_app="my-service",source_version="unknown",destination_workload="api-linter",destination_workload_namespace="my-namespace",destination_principal="unknown",destination_app="my-service",destination_version="unknown",destination_service="my-service",destination_service_name="my-service",destination_service_namespace="my-namespace",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="none",source_canonical_service="my-service",destination_canonical_service="api-linter",source_canonical_revision="latest",destination_canonical_revision="latest",request_method="GET"} 4204

** My question is, how can I add request path as a label to the main istio_requests_total metric like I did for request_method="GET" in the example above?**


Solution

  • The url_path and method are not part of the DefaultStatTags that get extracted automatically.

    To make it work, you must add the extraStatTags annotation (sidecar.istio.io/extraStatTags: request_path,request_method) to the deployment (or configure it mesh-wide via mesh config). See the example in the docs.

    Also, you should consider upgrading to the newer version of Istio as 1.8 was released in 2020 and not supported anymore.