Search code examples
monitoringrrdtoolganglia

Ganglia Metric Value Decay To Zero


I am sending metrics to Ganglia using Gmetric once every 5 minutes. But if the cron task that is sending the metrics is not working, I would expect the metric value on the Ganglia graph to go to zero, whereas it stays at its last updated value. Is there a way to change this behaviour?


Solution

  • Note that using the '-d' flag (aka DMAX) in your gmetric will make gmond/gmetad forget about the metric completely after DMAX seconds have passed without a gmetric update.

    So if you're running Ganglia straight out of the box, the .rrd file and the Ganglia graph for that metric will disappear entirely.

    If you'd like to preserve the .rrd history of the metric and/or you want the Ganglia graph to stay around (but have the plot drop to zero), you could edit the call to rrdtool inside /path/to/webserver/ganglia/graph.d/metric.php.

    Check out the rrdtool docs for CDEF and VDEF to see how you can incorporate conditionals into rrdtool calls. Within those statements I believe you can inspect the timestamp of the last metric inside the .rrd and then arbitrarily graph a value of zero (or any other value you want). Alternatively you could change the color or style of the line for time periods that you don't trust, etc., etc.

    rrdtool is fairly powerful, if incomprehensible. The best man pages to peruse would be rrdgraph_data, rrdgraph_graph, and rrdgraph. Also check out some of the other PHP scripts provided by Ganglia to see more examples of how to invoke rrdtool.