Trend analysis using iterative value increments

Question

We have configured iReport to generate the following graph:

The real data points are in blue, the trend line is green. The problems include:

• Too many data points for the trend line
• Trend line does not follow a Bezier curve (spline)

The source of the problem is with the incrementer class. The incrementer is provided with the data points iteratively. There does not appear to be a way to get the set of data. The code that calculates the trend line looks as follows:

import java.math.BigDecimal;
import net.sf.jasperreports.engine.fill.*;

/**
 * Used by an iReport variable to increment its average.
 */
public class MovingAverageIncrementer
  implements JRIncrementer {
  private BigDecimal average;

  private int incr = 0;

  /**
   * Instantiated by the MovingAverageIncrementerFactory class.
   */
  public MovingAverageIncrementer() {
  }

  /**
   * Returns the newly incremented value, which is calculated by averaging
   * the previous value from the previous call to this method.
   * 
   * @param jrFillVariable Unused.
   * @param object New data point to average.
   * @param abstractValueProvider Unused.
   * @return The newly incremented value.
   */
  public Object increment( JRFillVariable jrFillVariable, Object object, 
                           AbstractValueProvider abstractValueProvider ) {
    BigDecimal value = new BigDecimal( ( ( Number )object ).doubleValue() );

    // Average every 10 data points
    //
    if( incr % 10 == 0 ) {
      setAverage( ( value.add( getAverage() ).doubleValue() / 2.0 ) );
    }

    incr++;

    return getAverage();
  }


  /**
   * Changes the value that is the moving average.
   * @param average The new moving average value.
   */
  private void setAverage( BigDecimal average ) {
    this.average = average;
  }

  /**
   * Returns the current moving average average.
   * @return Value used for plotting on a report.
   */
  protected BigDecimal getAverage() {
    if( this.average == null ) {
      this.average = new BigDecimal( 0 );
    }

    return this.average;
  }

  /** Helper method. */    
  private void setAverage( double d ) {
    setAverage( new BigDecimal( d ) );
  }
}

How would you create a smoother and more accurate representation of the trend line?

Answer 1

This depends on the behavior of the item you are measuring. Is this something that moves (or changes) in a manner that can be modeled?

If the item is not expected to change, then your trend should be the underlying mean value of the entire sample set, not just the past two measurements. You can get this using Bayes theorem. The running average can be calculated incrementally using the simple formula

Mtn1 = (Mtn * N + x) / (N+1)

where x is the measurement at time t+1, Mtn1 is the mean a time t+1, Mtn is the mean at time t, and N is the number of measurements taken by time t.

If the item you are measuring fluctuates in a manner that can be predicted by some underlying equation, then you can use a Kalman filter to provide a best estimate of the next point based on the previous (recent) measurements and the equation that models the predicted behavior.

As a starting point, the Wikipedia entry on Bayesian estimators and Kalman Filters will be helpful.