Search code examples
prometheusmetricsistioenvoyproxy

Overwrite source_workload metric using istio operator


I would like to overwrite the source_workload tag in the request_bytes_bucket metric due to high cardinality issues generated by a specific workload - I do not want to drop the source_workload tag as that will break Kiali functionality and affect other workloads that do not have this issue - I've tried using the following definition but this does not seem to be working, there doesnt appear to be many examples so I'm not sure if a ternary or regex can be used in this scenario:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio
spec:
  values:
    telemetry:
      v2:
        prometheus:
          configOverride:
            inboundSidecar:
              metrics:
                - name: request_bytes_bucket
                  dimensions:
                    source_workload: "matches(istio_request_bytes_bucket.source_workload, \"source-workload-prefix-.*\") ? istio_request_bytes_bucket.source_app : istio_request_bytes_bucket.source_workload"

Solution

  • To get this working I had to drop the 'istio' prefix, the 'bucket' suffix and add an additional set of single quotes around the value e.g.

    values:
       telemetry:
         enabled: true
         v2:
           enabled: true
           prometheus:
             configOverride:
               inboundSidecar:
                 stat_prefix: istio
                 metrics:
                   - name: request_bytes
                     dimensions:
                      source_workload: "'test'"