Search code examples
javaweka

IndexOutOfBoundsException when trying to add more instances to training set using Weka

I am trying to add more Instances to my training set and perform 10-fold cross validation.

My instances are in String format so i use the StringToWordVector filter to transform them to numbers. Things work well if i do not add the extra pages i want. But when i add the command trainSet.addAll(data2); and pass trainSet to the filter i get a strange IndexOutOfBoundsException in the first iteration at Instances fTrainSet = Filter.useFilter(trainSet, filter);

Instances data = getDataFromFile("pathtofile.arff");//main dataset 1821 instances
Instances data2 = getDataFromFile("anotherpath.arff");//709 instances i want to add 
int folds = 10;
for(int i=0;i<folds;i++){
    Instances trainSet = data.trainCV(folds, i);//training set
    System.out.println(trainSet.numInstances());//Prints 1638
    Instances testSet =  data.testCV(folds, i);//testing set

    //add more instances
    trainSet.addAll(data2);        
    System.out.println(trainSet.numInstances());//Prints 2347

    //filter
    StringToWordVector filter = new StringToWordVector();
    filter.setInputFormat(trainSet);        
    filter.setWordsToKeep(10000);
    filter.setTFTransform(true);
    filter.setLowerCaseTokens(true);
    filter.setOutputWordCounts(true);
    Stemmer stemmer = new IteratedLovinsStemmer();
    filter.setStemmer(stemmer);
    WordsFromFile stopwords = new WordsFromFile();
    stopwords.setStopwords(new File(".data/stopwords2.txt"));
    filter.setStopwordsHandler(stopwords);

    Instances fTrainSet = Filter.useFilter(trainSet, filter);//error!!!
    Instances fTestSet = Filter.useFilter(testSet, filter);
    ....
    //classification and evaluation....

I get the following error when i am trying to use the filter:

Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 2161, Size: 1749
    at java.util.ArrayList.rangeCheck(Unknown Source)
    at java.util.ArrayList.get(Unknown Source)
    at weka.core.Attribute.addStringValue(Attribute.java:924)
    at weka.core.StringLocator.copyStringValues(StringLocator.java:150)
    at weka.core.StringLocator.copyStringValues(StringLocator.java:91)
    at weka.filters.Filter.copyValues(Filter.java:399)
    at weka.filters.Filter.bufferInput(Filter.java:342)
    at weka.filters.unsupervised.attribute.StringToWordVector.input(StringToWordVector.java:655)
    at weka.filters.Filter.useFilter(Filter.java:692)
    at CrossValidationExample.main(CrossValidationExample.java:108)

What could be wrong?

Solution

  • After some searching i realize that there is something wrong with the addAll function. One reason i can think of is that addAll just adds references of instances and that is an issue when i try to use them with the filter . Instead, i used the merge function proposed here https://stackoverflow.com/a/12359788/3923800 ,so i replaced trainSet.addAll(data2); with Instances newTrainSettrainSet = merge(trainSet,data2); and everything works fine.