Search code examples
csvdygraphsstandard-deviation

Ignore zero values in dygraphs when standard deviation is above a certain value

I'm using dygraphs to display weather data collected every ten minutes. One of the datapoints is snow depth (in meters). Once in a while the depth is wrong, 0 meters, where the previous and next is 0.9 meters. It's winter atm. and I've been on the location to verify 0.9 is correct.

With 47 datapoints at 0.9 m. and one at 0 m. standard deviation is approx. 0.13 (using ministat on FreeBSD).

I've looked through the dygraphs documentation but can't find a way to ignore values like 0 when the standard deviation is above a certain threshold.

This page on dygraphs have three examples on how to deal with standard deviation but I just want to ignore the 0, not use it with the option errorBar or customBar, and the data is not in fractions.

The option rollPeriod is not applicable since it merely averages x data points.

I fetch the weather data in xml-format from a third party every ten minutes in a cron job, parse the values and store them in postgresql. Then select the last 48 data points in another cron job and redirect the data to a csv-file which dygraphs consumes.

Can I have dygraphs ignore 0 if standard deviation is above a threshold?

I can get the standard deviation with ministat or other utility from the last cron job and remove the 0 from the cvs-file using sed/awk. But will prefer dygraphs to do that.

var g5 = new Dygraph(
    document.getElementById("snow_depth"),
    "snow.csv",
    {
        legend: 'always',
        title: 'Snødjupne',
        labels: ["", "djupne"],
        ylabel: 'Meter'
    }
);

First three lines of the csv-file.

2016-02-23 01:50:00+00,0.91
2016-02-23 02:00:00+00,0
2016-02-23 02:10:00+00,0.9

enter image description here

Solution

  • You should clean your data before you chart it! dygraphs is a charting tool, not a data processing library.

    The best approach is to load your data via AJAX, parse it, filter out the zeros and feed it in to dygraphs using native format.