I have a metric nomad_raft_appliedIndex with env, job and instance labels (and also some other labels). The metric is increasing every interval by some value. The increase itself is not important - the difference of increase between instances is important.
What I want to do, is to take minimum every interval and subtract for every instance. So assuming values like (time, instance1, instance2):
2022-12-09 11:32:30 1040095 1040095
2022-12-09 11:32:45 1040102 1040100
2022-12-09 11:33:00 1040103 1040106
I would want to get:
2022-12-09 11:32:30 0 0
2022-12-09 11:32:45 2 0
2022-12-09 11:33:00 0 3
How do I do that? I am fine with any result that will show relative differences between values. So increase or rate but relative to the smallest or highest value in particular interval showed by grafana. I've tried various combinations of operators:
nomad_raft_appliedIndex{env="$env",job="nomad"}
# -
#on (env, job)
#group_left (instance)
#(min by (env) without (instance)
#(bottomk by (env, job)
#(1, nomad_raft_appliedIndex{env="$env",job="nomad"}))
However none of them are working. Examples:
The following:
nomad_raft_appliedIndex{env="$env",job="nomad"}
- on (env, job) group_left (instance) min without(instance) (nomad_raft_appliedIndex{env="$env",job="nomad"})
Results in:
execution: multiple matches for labels: grouping labels must ensure unique matches
The following:
nomad_raft_appliedIndex{env="$env",job="nomad"}
- on (env, job)
(bottomk by (env, job)
(1, nomad_raft_appliedIndex{env="$env",job="nomad"}))
Results in:
execution: multiple matches for labels: many-to-one matching must be explicit (group_left/group_right)
The following:
nomad_raft_appliedIndex{env="$env",job="nomad"}
- (bottomk by (env, job)
(1, nomad_raft_appliedIndex{env="$env",job="nomad"}))
Results in only one instance with all zeros.
The following:
nomad_raft_appliedIndex{env="$env",job="nomad"}
- (min without(instance) (nomad_raft_appliedIndex{env="$env",job="nomad"}))
Results in "No data" in grafana, which I read it means that "labels do not match".
I tried also sprinkling the code with [$__interval] but with no success.
The following combination works:
nomad_raft_appliedIndex{env="$env",job="nomad"}
- on (env, job) group_left()
(min by(env, job) (nomad_raft_appliedIndex{env="$env",job="nomad"}))