Search code examples
prometheusgrafana

Grafana interval variable: minimum interval threshold function?

I'm adding an interval variable to a dashboard to allow averaging some prometheus time series over an interval to get cleaner looking graphs. But I want to leave in the base sample rate (15s) so the graph can be viewed without averaging.

However, some graphs use rate() which gives no data for the base sample rate.

How can I share $interval for all time series in the dashboard, but force graphs using rate() to use a minimum interval (e.g., 1m) ? Is there something I can add to the query like max($interval,1m)

Just to add a little more detail...the data for the time series for one of the graphs looks like: 100 0 100 0 100 0 ... and I want to average that to 50 (they are not all that clean) by using a time interval larger than $__interval.

I suppose what I need to do is use a grafana interval variable (call it $myinterval) which I already have and make a chain variable $myinterval_rate but I'm not sure how to calculate it in grafana.

Solution

  • You can create custom variable with key : value pairs, and use its "keys" for "averaging" functions, and "values" for rate-related functions.

    This can be done in the following manner:

    1. Create custom variable with following values:
    15s : 1m, 30s : 1m15s, 1m : 2m, 2m : 2m15s

    Notice spaces surrounding colon, they are important for values parsing. Also, notice, that values of different pairs should be different.

    1. Go to your queries:

    2.1. For those that use you averaging use variable in the form ${myvar:text}, where myvar is the name of created variable. This will substitute "key" of the variable. For first pair in my example: 15s.

    2.2. For queries that use rate-like functions use variable in the form ${myvar:value}. This will substitute "value" of the variable. For first pair in my example: 1m.

    If uniqueness of values for the variable is a deal breaker for you, you can use chaining of the variables.

    For this case query will look something like this:

    query_result(
  (absent(non_existent{pseudo_label="1m"}) and on() (absent(non_existent{pseudo_input="15s"}) and absent(non_existent{pseudo_input="$myvar"}))) or
  (absent(non_existent{pseudo_label="1m"}) and on() (absent(non_existent{pseudo_input="30s"}) and absent(non_existent{pseudo_input="$myvar"}))) or
  (absent(non_existent{pseudo_label="1m"}) and on() (absent(non_existent{pseudo_input="1m"}) and absent(non_existent{pseudo_input="$myvar"}))) or
  (absent(non_existent{pseudo_label="2m"}) and on() (absent(non_existent{pseudo_input="2m"}) and absent(non_existent{pseudo_input="$myvar"})))

    where $myvar is a variable with your initial intervals.