C++/OpenCV: How to use BOWImgDescriptorExtractor to determine which clusters relate to which images in the vocabulary?

Question

My goal is to take an image as a query and find its best match in a library of images. I am using SURF features, in openCV 3.0.0, and the Bag of Words approach to find a match. I need a way to find out if the query image has a match in the library. If it does, I want to know the index of the image that is the nearest match.

Here is my code for reading in all the images (300 total in the library of images) and extracting and clustering the features:

Mat training_descriptors(1, extractor->descriptorSize(), extractor->descriptorType());
//read in all images and set to binary
char filepath[1000];
for (int i = 1; i < trainingSetSize; i++){
    cout << "in for loop, iteration: " << i << endl;
    _snprintf_s(filepath, 100, "C:/Users/Randal/Desktop/TestCase1Training/%d.bmp", i);
    Mat temp = imread(filepath, CV_LOAD_IMAGE_GRAYSCALE);
    Mat tempBW;
    adaptiveThreshold(temp, tempBW, 255, ADAPTIVE_THRESH_GAUSSIAN_C, THRESH_BINARY, 11, 2);
    detector->detect(tempBW, keypoints1);
    extractor->compute(tempBW, keypoints1, descriptors1);
    training_descriptors.push_back(descriptors1);
    cout << "descriptors added" << endl;

}
cout << "Total descriptors: " << training_descriptors.rows << endl;
trainer.add(training_descriptors);

Ptr<DescriptorMatcher> matcher = DescriptorMatcher::create("FlannBased");
BOWImgDescriptorExtractor BOW(extractor, matcher);
Mat library = trainer.cluster();
BOW.setVocabulary(library);

I wrote the following code in an attempt to find a match. The problem is that BOW.compute only returns the indices of clusters (words) that exist in both the image and the library of images. imgQ is the query image.

Mat output;
Mat imgQBW;
adaptiveThreshold(imgQ, imgQBW, 255, ADAPTIVE_THRESH_GAUSSIAN_C, THRESH_BINARY, 11, 2);
imshow("query image", imgQBW);
detector->detect(imgQBW, keypoints2);
extractor->compute(imgQBW, keypoints2, descriptors2);

BOW.compute(imgQBW, keypoints1, output);
cout << output.row(0) << endl;

I need to know which clusters in the BoW correspond to which images. My output right now--output.row(0)--is just an array with all the the indices of clusters found in the library. Am I misunderstanding this output? Is there a way to determine which image has the most matching clusters?

Answer 1

I also did something similar based on this code:

https://github.com/royshil/FoodcamClassifier/blob/master/training_common.cpp

But the above part is after the clustering has finished. What you have to do is to train using your ML(I used SVM) and your cluster centers, the visual bag of words that you have. More, you need to find all the "closest" points to your clustered points and train them using histograms. Next, you will have a histogram of frequencies(bag of key points) that you need to train.

Ptr<ifstream> ifs(new ifstream("training.txt"));
int total_samples_in_file = 0;
vector<string> classes_names;
vector<string> lines; 

//read from the file - ifs and put into a vector
for(int i=0;i<lines.size();i++) {

    vector<KeyPoint> keypoints;
    Mat response_hist;
    Mat img;
    string filepath;

    string line(lines[i]);
    istringstream iss(line);

    iss >> filepath;

    string class_to_train; 
    iss >> class_to_train; 
    class_ml = "class_" + class_to_train;
    if(class_ml.size() == 0) continue;

    img = imread(filepath);

    detector->detect(img,keypoints);
    bowide.compute(img, keypoints, response_hist);

    cout << "."; cout.flush();
    //here create the logic for the class to train(class_0, e.g) and the data you need to train.
}

More you can find at this git project:
https://github.com/royshil/FoodcamClassifier
Documentation here:
http://www.morethantechnical.com/2011/08/25/a-simple-object-classifier-with-bag-of-words-using-opencv-2-3-w-code/