I am trying to set up an alert in the alert.rules file of Prometheus that fires only during specific periods of time.
I've already tested the code block below without time restrictions inside the expr-tag and it works absolutely fine.
As PromQL Documentation: hour() states, hour()
returns a value between 0 and 23 depending on the current UTC.
- alert: test_down
expr: absent(container_memory_usage_bytes{name="test_ap"}) and hour() > 5 and hour() < 22
for: 30s
labels:
severity: critical
annotations:
summary: "test_ap down"
description: "test_ap is down for more than 30 seconds."
But here, no alert notification is fired. Does anybody know, why nothing is fired and how I can fix that?
EDIT: I already solved it. I don't understand why I have to do it like the way I am doing it, but the following works:
replace and hour() > 5 and hour() < 22
with and ON() hour() > 5 < 22
In this case, ON() it's join operation, which ignores matching labels from left part. Otherwise, Prometheus would expect the same set of labels on left and right side. you may read more in this blogpost.