Search code examples
prometheus

Can prometheus relabeling section relabel_config apply to timeseries/metrics


Prometheus documentation says that relabel_config applies before scrape. However reading through some docs (attached), It mentions that relabel_config can be used to drop target or series.

If relabel_config is section is applied before scrape, how is it possible to have series before scrape?

Ref: https://grafana.com/docs/grafana-cloud/account-management/billing-and-usage/control-prometheus-metrics-usage/usage-reduction/?pg=blog&plcmt=body-txt

enter image description here

Another reference which is confusing as well (The example shows that relabel_config block how we can operate on the metrics to change label values) enter image description here Ref: https://grafana.com/blog/2022/03/21/how-relabeling-in-prometheus-works/#the-base-relabel_config-block


Solution

  • It seems like there is a bit of confusion cause by similar but not the same terms.

    I believe that by this

    Prometheus documentation says that relabel_config applies before scrape

    you mean following line from official Prometheus documentation:

    Relabeling is a powerful tool to dynamically rewrite the label set of a target before it gets scraped.

    This parts refer to <relabel_config> section of Prometheus' configuration, and operates over labels associated with target: job, __address__, __metrics_path__ and a couple more.

    This configuration in fact allows for targets to be dropped altogether from scraping even before scraping happened.


    Relabel_config mentioned in linked material, on the other hand, refers to family of relabel_configs present in Prometheus: <relabel_config>, <metric_relabel_configs> and <alert_relabel_configs>.

    And since it describes abstraction over those configs with different targets, it uses mentioned "target or series" phrase to underscore fact, that targets differ depending on exact stage.

    Some more information on difference between those relabeling stages can be found in the same material here.