I'm trying to subtract two metrics (guages) with the same lables over a time period [5m], and then average the result.
Queries I have tried,
avg_over_time(jvm_memory_bytes_max{area="heap", app="ui",kubernetes_pod_name="ui-dep-76b4f95bf4-xcs4s"}[5m] - jvm_memory_bytes_used{area="heap", app="ui",kubernetes_pod_name="ui-76b4f95bf4-xcs4s"}[5m])
avg_over_time(jvm_memory_bytes_max{area="heap", app="ui",kubernetes_pod_name="ui-dep-76b4f95bf4-xcs4s"} - jvm_memory_bytes_used{area="heap", app="ui",kubernetes_pod_name="ui-76b4f95bf4-xcs4s"})[5m]
avg_over_time((jvm_memory_bytes_max - on(area, app, kubernetes_pod_name) jvm_memory_bytes_used)[5m])
All of these have parse errors because the query is not correct.
Most of the example over ranges use avg_over_time to reduce a metric to a single value and then use an arithmetic operation.
I'm not sure if it's applicable because I don't want to average over a metric and then apply an arithmetic function. But rather, I want to apply the arithmetic operation on two data points of different metrics and then average the results.
Is that possible ? If so, how would I go about it ?
Try the following query:
avg_over_time((jvm_memory_bytes_max{area="heap", app="ui",kubernetes_pod_name="ui-dep-76b4f95bf4-xcs4s"} - jvm_memory_bytes_used{area="heap", app="ui",kubernetes_pod_name="ui-76b4f95bf4-xcs4s"})[5m:10s])
It calculates the difference of two metrics on every 10-second interval and then calculates the average over the differences for the last 5 minutes. The query uses Prometheus subquery feature.