I have set up some alerts using the in-house grafana alertmanager and so far that is working fine.
I'm monitoring temperatures in a lab and have set the alertmanager to send me an alert, if the temperature of a sensor is > 32 degrees celsius.
The only problem I now have is that there is no option to configure hysteresis / a resolved condition for the alert. So currently, if the temperature is > 32 degrees, an alert is sent to me via email. So lets say for example the temperature is 32.1, I receive an alert and the shortly after the temperature drops back down to 32 degrees and I receive a resolved notification, but then just minutes later the temperature goes up back to 32.1 and I receive an alert again etc etc etc.
My question now is, can I somehow set a condition for when the alert gets resolved? For example I only receive a resolved notification if the temperature drops to 31.5.
Does anyone know if something like that is possible?
I tried to set a hysteresis but there is no option for that. Or atleast I haven't found it yet.
Grafana Alerts provide "Pending Period" setting that is a limited form of hysteresis. When a value exceeds the threshold, it enters the pending state. This does not generate an alert until the pending period expires. After this, if the condition is still breached, the alert is fired. While this is not full featured hysteresis, it can somewhat help with the "noise" at the expense of latency.