I am monitoring disk space usage using metricbeat.
Now I want alerts via Elastalert depending on the disk size.
Alert when disk space used crossed 50%
Alert when disk space used crossed 70%
Alert when disk space used crossed 80%
Alert when disk space used crossed 95%
Alert when disk space used crossed 100%
Now the catch here is that alerts should be raised only once when it crosses certain thresholds (50, 70, 80, 95, 100)
So, if alert is already sent for crossing 50% mark it should not send alert for 50.1% / 50.2% / ... / 69.9%
The next alert should only be raised when it crosses 70%.
Initial Approach:
If (dir size==50 || dir size==70 || dir size ==80 || dir size==95 || dir size ==100)
alert
I planned to use "any rule" to match disk space field to different values and alert. But this may generate false alerts too, reason being if the storage is saturated at 50.0% (consider no new data written to DB) for the last 1 hour and if we evaluate rules every 10 mins, it will raise alert 6 times in that hour. Also I don't want to use realert as I don't know for how long to wait.
Approach v1:
Make n number of rule configs where n is the number of different conditions use a realert setting that is so long it's effectively "never"
realert:
weeeks: 9999
This approach is not ideal as we need repeated alerts. Example - When usage drops below 50% and then crosses 50% again, Alert is required.
Approach v2: Combination of two rule can be used. (consider for 50% only)
Rule 1: check disk space >= 50, send mail, enable rule 2 and disable itself using command
Rule 2: check disk space <50, enable Rule 1, disable itself using command.
Any better approach?
Created a custom rule. For more details check this post: Using Elastalert to monitor disk growth