I am trying to create a PromQL query where I can output metrics in a dynamic time interval.
I monitor a CLI tool with custom metrics via Prometheus. The tool is run a few times a day. During runtime metrics like number of processed objects or time of processing are collected.
Now I want to create a dashboard in Grafana, where I can see for each execution of the CLI tool in a bar chart or similar, how long the last executions took and how many objects were processed.
If you look at the gauge in a bar chart, you can see the different executions each with the maximum value for that cycle. That is the value I am looking for.
Now I can aggregate the values and summarize them in 15 minute intervals. Theoretically this looks like the solution, but it only works if exactly one execution takes place in a 15 minute interval. As soon as a second execution takes place in the same interval, it comes to wrong values. And if one execution exceeds the interval, it looks like two executions.
Generally, I can define the time period with metrics or by changing the labels of metrics.
I've found a solution to my problem. Through the Micrometer Observation API, which is available in Spring Boot 3, observation events can be sent as metrics. These events are stored as time series with the value 1 in Prometheus. If I join the two metrics and match the values by the group modifier to the event, I get the value I am looking for at the time of the event.
observation.event(Observation.Event.of("finished"));
execution_duration_seconds_max{application="xyz",type="xyz"} * on () group_right() execution_finished_total