Relabelling in prometheus

Question

I am trying to relabel after copying the existing metrics for Kafka only for the below match:

Kafka_log_size{partition=“1”, topic=“ab_bc_cd_12345_ef_001”,} 10

I want output as:

Kafka_log_size{partition=“1”, topic=“ab_bc_cd_12345_ef_001”,} 10
Kafka_log_size_uniq{partition=“1”, uniq=“12345”,} 10

From reading multiple articles, it seems this can be achieved by relabeling but I'm not sure how to start.

Answer 1

You have two options to relabel your metrics in Prometheus:

• at ingestion time using metric_relabel_configs (in Prometheus configuration)
• at request time using label_replace

In both cases, you need to have a regex that matches the content of topic label and extracts what you need.

Relabeling in config

In your job scraping for Kafka, you have to match the metric name you want to replace and use relabeling config:

metric_relabel_configs:
  - source_labels: [topic] 
    regex: '[a-z_]+_([0-9]+)_.*'
    action: replace
    target_label: uniq
    replacement: $1

And then add a rule to drop the label

  - regex: 'topic'
    action: droplabel

Relabeling in request

The new label is generated using label replace:

label_replace(Kafka_log_size, "uniq", "$1", "topic", "[a-z_]+_([0-9]+)_.*")

And then, apply an aggregation operator to remove the unwanted label:

max(label_replace(Kafka_log_size, "uniq", "$1", "topic", "[a-z_]+_([0-9]+)_.*")) without(topic)