I have a custom Gauge sensor_status
with labels job, type, id.
I want to get the time when sensor_status{job='myjob', type='streaming'} == 0
for more than 10 minutes.
The result should be data with timestamp when it occurs by each label id.
sensor_status{job='myjob', type='streaming'} == 0 and (changes(sensor_status{job='myjob', type='streaming'}[10m]) == 0)
I have tried the query above but seem like it returns random time (maybe at the time near time of missing value).
And when the value is equal to 0 for 1 hours, it will return 6 time intervals. Is there a better way to check if it's equal 0 more than 10 minutes and return the whole big interval?
I'm sending the query through API /api/v1/query_range
btw.
Your query is correct for what you described.
If you want to only get "last" 10-minutes window where your metric is 0 for 10 minutes, and recent 10 minutes are not critical for you, you can add this condition to the query:
and changes((sensor_status offset -10m)[10m:])!=0
Or if recent 10 minutes are critical, you may use something like
and changes((sensor_status offset -10m)[10m:])!=0 or absent(sensor_status offset -10m)
A couple little side notes based on your question.
First of all, it feels like what you are trying to achieve is kind of alerting based on lack of data. If this is the case, I strongly suggest moving to built-in Prometheus' alerting.
Second. Based on this quote
And when the value is equal to 0 for 1 hours, it will return 6 time intervals
I assume that you're using step equal to 10 minutes. Beware, as it can result in loss of desired/expected data: if 10 minutes of zero-valued window will be split between two time-windows (of step length), query would not return result. You'll have a result only if query returns result on the end of step.