Incosistency of results in ten-fold cross validation in WEKA

Question

Yesterday, I use 2 ways to implement the 10-fold cross validation in weka, but the results are inconsistent.

Way 1: directly invoke method eval.crossValidateModel(),

 J48 j48 = new J48();
 j48.buildClassifier(ins);  // ins is the Instances object
 Evaluation eval = new Evaluation(ins);
 eval.crossValidateModel(j48, ins, 10, new Random(1)); // 10-fold cross validation
 ... // get results by eval.getXX(0) or eval.getXXX(1)

Way 2: using method testCV() and trainCV() in each fold,

 ins.randomize(new Random(1)); // ins is the Instances object
 ins.stratify(10); // randomize the dataset then split into 10 folds

 for(int i=0; i<10; i++){       
    Instances trainData = ins.trainCV(10, i);
    Instances testData = ins.testCV(10, i);
    J48 j48 = new J48();
    j48.buildClassifier(trainData);

    Evaluation eval = new Evaluation(trainData);
    eval.evaluateModel(j48, testData);
    ... // get results by eval.getXX(0) or eval.getXXX(1)
 }

According to the weka api docs, the above 2 ways should have the same results, i.e., the average results(e.g., precision, recall) of way 2 should be equal to the results of way 1. But the truth is that they are not the same, can anybody figure out the bug in my code, or provide other nice evaluation methods? Thank you all!

Answer 1

If you look at the code of the weka.classifiers.Evaluation.crossValidateModel method (depending on your version, the delegate object), you will see that it uses the weka.core.Instances.trainCV(int,int,Random) method. Furthermore, you need to initialize the Evaluation object with the class priors of the complete dataset.

Here's the updated code:

Evaluation eval = new Evaluation(ins);  // init evaluation
rand = new Random(1);
int numFolds = 10;  // 10-fold CV
ins.randomize(rand); // randomize the data
ins.stratify(numFolds); // stratify the randomized data for 10-fold CV
J48 template = new J48();  // classifier template for evaluation
//template.setOptions(...);  // if further options need to be set

for (int i = 0; i < numFolds; i++) {       
  Instances trainData = ins.trainCV(numFolds, i, rand);
  Instances testData = ins.testCV(numFolds, i);
  Classifier cls = AbstractClassifier.makeCopy(template);  // copy of classifier template
  cls.buildClassifier(trainData);
  eval.evaluateModel(cls, testData);  // accumulate statistics
}

... // get results by eval.getXX(0) or eval.getXXX(1)