I have a metrics of gauge type that, for each action (label step) and a give date (label day in the form 'yyyy-mm-dd'), stores the timestamp of the last succeeded event. Something like:
{step_name="Upload table_1", day="2022-01-01"} = 1644354491
I can, for instance, check how many hours ago data of day "2022-01-01" was uploaded in "table_1" with:
(time() - importer_succeeded_timestamp{day = "2022-01-01", step_name="Upload table_1"})/3600
I need to be alerted if the day after data wasn't uploaded in a table, so if for a difference greater than X of the value of the gauge and the converted date-value of "day" label, the metric is not present. So somehow I need to compare the value of the gauge (timestamp) with the "day" (date in string format). Every help is very welcome, my experience with Prometheus metrics and alerting manager is quite limited but I have the sensation that with the current metrics design is not possible.
I continued to search and what I could figure out is that it's not possible.
In this conversation on Grafana blog https://community.grafana.com/t/how-to-use-a-prometheus-label-as-a-timestamp-for-grafana/54708 they speak about a:
known limitation with Prometheus. It just wasn’t designed to ingest time-series data where the timestamp is a label.
Here another question similar to mine Prometheus label comparison with vector timestamp
This isn't possible, label values are opaque strings to Prometheus.
The only solution we've found: adding a new metrics with the same set of labels, having as value the seconds from the the "day label" to the moment of the execution. Like this it's possible to use both the metrics in AND, because how it's written here https://prometheus.io/docs/prometheus/latest/querying/operators/#logical-set-binary-operators:
vector1 and vector2 results in a vector consisting of the elements of vector1 for which there are elements in vector2 with exactly matching label sets. Other elements are dropped. The metric name and values are carried over from the left-hand side vector.