Search code examples
time-seriesprometheusgrafanamonitoringmetrics

Fill Missing Values in Time Series in Prometheus

I have a metric that has multiple time series in Prometheus. I want to fill in the gaps in this metric with a default value (say 0). What's the best way to do this? ORing with vector(0) doesn't work as there are multiple time series - All Prometheus does is give me a new time series that is always 0. I don't want to use recording rules either.

pinot_server_llcPartitionConsuming_Value{} OR vector(0) results in the following:

enter image description here

Solution

  • To fill in gap, that is no longer than 1 hour, you can use query like

    metric or last_over_time(metric [1h]) * 0

    This query will return metric if it's found. And if not, it will take latest to the moment sample of metric (to simply preserve all the labels) and return it multiplied by 0.

    Please be aware, that shown query doesn't actually check for "gap". It will keep producing values for one more hour even if metric "died out" (due to target in unavailability, for example).

    If you actually need to check for gaps specifically, you'll need to implement a check for that around substitution. It might look something like

    metric 
 or (
  last_over_time(metric [1h]) * 0
  and last_over_time(timestamp(up)[1h:1m] offset -1h) > time()
 )

    Here substitution with 0 happens only if metric was seen reported again within hour from timestamp for which possible substitution is calculated.