Dropwizard metrics (Meter) 3.1.0 + InfluxDB 0.9.x + Grafana stuck at the latest value

Question

As the title suggests, I am using the above stack for insight into the frequency of a particular method call over time.

This is how collecting metrics is set up in the code:

 public void searchProducts(Query query) {
    Meter meter = metricRegistry.meter(MetricRegistry.name("search.products"));
    meter.mark();

    // perform search
    // ...
  }

This is how the graph is configured in Grafana:

And this is how the resulting graph looks like:

As you can see, when there are no method calls, the line is stuck at the latest value. There are several drops to 0 (right after two big 'humps' at the beginning and one more at the end) which happened when I stopped the application.

Shouldn't the line drop to 0 after some time passes after the last mark? E.g. the application is started, several method calls occur, 1 second passes between each method call and then after about 30 seconds, the line should go back down to 0 and stay there. Like this:

• 00:00 start of the application (line is at 0)
• 00:01 method call (line goes to 1)
• 00:02 method call (line goes to 2)
• 00:03 method call (line goes to 3)
• 00:04 nothing happens (line stays at 3)
• 00:05 nothing happens (line stays at 3)
• ...
• 00:30 nothing happens (line drops to 0)

Instead, Meter in this case seems to behave just like the Counter - incrementing values for the whole lifecycle of the application.

EDIT: Here is how the graph looks like even with Display styles -> null point mode -> null as zero is set. Graph looks exactly the same with Display styles -> null point mode -> null setting, as well.

UPDATE: Sorry for the delay in response - only recently I came back to this problem. I found out the root cause of the problem - it was that underlying database was InfluxDB with custom Metrics Reporters (there are no official ones for InfluxDB) and there was a bug in the reporter logic.

Thanks everyone for the help.

Answer 1

For graphs, there is an option under the Display Styles tab where you can choose how to handle null values. If connected is chosen, then you will see the behavior you describe. There are two other options, null -> no line or null as zero -> line drops to zero.

On older versions of Grafana, the option is call Null point mode (docs here)

Here is a screenshot of the option in Grafana 3.1: