Search code examples
spring-bootgrafanagraphitespring-boot-actuator

Grafana singlestat and transformNull

I'm using Grafana 5.0.3 with a Graphite data source and I'm having some issues with Singlestat using a transformNull function. The data source referenced below is from Spring Boot 2 Micrometer's http metrics, so I can't modify the code that generates this data.

I have a graph that tracks HTTP 500 errors since the last server restart.

Scenario 1 - Data Becomes Null

This happens when I restart the server and no new HTTP 500 errors have occurred: graph with nulls

I also have a Singlestat showing the current value of that graph. If I use transformNull to convert null data to 0s it works as expected (transformNull before or after sumSeries shows the same result): singlestat 1 with tranformnull

If I omit transformNull then I seemingly get whatever the sum was at the last time data was collected, regardless of the fact that at the current time the value should be 0: enter image description here

Scenario 1 Resolution

I should use transformNull because that will account for the current data's being null.

Scenario 2 - Data is Non Null

This happens after errors have occurred and the server is still running: enter image description here

If my Singlestat does not contain the transformNull function then it works as expected. I imagine this is because it shows the data at the last point in time data was collected: enter image description here

But, if I include transformNull, I see the data go to zero fairly frequently. My guess is that because data is being pulled from Graphite, there are times when Grafana is drawing the graph up until "now" but the Graphite data for "now" doesn't exist. My Spring Boot server pushes data to Graphite every 15s. enter image description here

Scenario 2 Resolution

I should not use transformNull because that will ensure to get the data from the last time data was collected.

Question

How do I go about accurately displaying this data in the Singlestat? Is it just some magic combination of the rate I'm pushing data from Spring Boot to Graphite and the Grafana refresh interval?

Solution

  • I don't know about any underlying solution but here's what worked for me. I used transformNull to ensure that the most recent sample has the correct numbers, then I changed the time range so it's not realtime but delayed by 30 seconds or a minute. E.g. for "Last 1 hour" I changed the window from "From: now-1h, To: now" to "From: now-1h, To: now-30s."

    It doesn't solve the problem but it's a workaround that's good enough for me.