I collect some solar power data in a prometheus database. Next I visualize my data using grafana.
I have created a dashboard with two display widgets, a timeseries and a stat with current value. The timeseries is good with a nice graph for the selected time period.
But the "current value" stats display isn't working quite as expected. I have the following query:
last_over_time(victron_mppt_batteryVoltage[1m])
Problem is that the selected time period affects the value shown in the stats widget. Clearly some aggregation is going on here.
How to get grafana to query prometheus for the "actual value" display without any aggregation going on?
By default Grafana queries Prometheus to get series of data, according to time range selected for the dashboard and step (or resolution) calculated.
So, let's assume you selected for your dashboard time range of last 5 minutes, and Grafana decided on step of 10 seconds.
This means that Prometheus will return 30 data points for each time series. Basically, for each 10 seconds of those 5 minutes query will be evaluated separately, and you'll get a vector of results associated with time for which they where evaluated.
After that Grafana will need to aggregate this data in some what you show you a single value. I think, default aggregation for Stat panel in Grafana is "Last *" (last non null), but it might be that in your case it's something else.
But even if you'll select "Last *", it will be not the best solution.
You can switch Type of the query to Instant (Query options are a collapsible panel under the query editor). This will switch Grafana from using api/v1/query_range to api/v1/query, and Grafana will automatically query Prometheus only for the most recent data point.
If you tried to use last_over_time to mediate your problem - remove it. Most likely it doesn't do what you expect it to.
last_over_time "stretches" value for metrics that are no longer there. So in your case, even if metric stopped being reported half an hour ago, it still will be return in a result.