Find a fragment in the whole image

Question

Globally, my task is to determine the similarity / dissimilarity of two .jpg files. Below I will describe the process in more detail. I have five (in reality there are more) template .jpg files. And I have a new .jpg file, which I must match with each template .jpg file to make a decision - is the new .jpg file similar to any of the template .jpg files or not.

Correlating entire files in my case is a bad idea, since the error is large. So I came up with a way to "cut" the new file into 12 equal parts (fragments) (that is, into 12 .jpg files) and search for each individual fragment in the template.

For this, I used the tutorial https://docs.opencv.org/4.x/dc/dc3/tutorial_py_matcher.html

But the problem is that the fragments from the new .jpg file are extremely incorrectly matched with the template.

Below I will show an example: Let's take the document below as a template

And the document below as a new document (schematically I cut it into 12 parts, that is, I receive one document as input, but I cut this one document into 12 parts (fragments) (that is, 12 new files))

Next, take a look at my code. The gist of it is that I take each of the 12 fragments and search for that fragment in the template

def match_slices_in_template(path_template):
    directory_in_str = 'slices'
    directory = os.fsencode(directory_in_str)

    img1 = cv.imread(path_template, cv.IMREAD_GRAYSCALE)  # queryImage

    good = []
    for slice_image in os.listdir(directory):
        print(slice_image)
        filename = os.fsdecode(slice_image)
        img2 = cv.imread(f'slices/{filename}', cv.IMREAD_GRAYSCALE)  # trainImage

        # Initiate SIFT detector
        sift = cv.SIFT_create()

        # find the keypoints and descriptors with SIFT
        kp1, des1 = sift.detectAndCompute(img1, None)
        kp2, des2 = sift.detectAndCompute(img2, None)

        # BFMatcher with default params
        bf = cv.BFMatcher()
        matches = bf.knnMatch(des1, des2, k=2)

        # Apply ratio test
        for m, n in matches:
            if m.distance < 0.3 * n.distance:
                good.append([m])

        # cv.drawMatchesKnn expects list of lists as matches.
        img3 = cv.drawMatchesKnn(img1, kp1, img2, kp2, good, None, flags=cv.DrawMatchesFlags_NOT_DRAW_SINGLE_POINTS)

        plt.imshow(img3), plt.show()

print(match_slices_in_template('Bill_1.jpg'))

But the search result is completely incorrect, take a look at some sample graphs that matplotlib built

In my example, in fact, the two files are different (although they have a lot in common). But the program identifies them as quite similar.

So you probably understand the essence of my question: how to improve the performance of the algorithm so that it would determine similarity/difference more accurately than now

Answer 1

I wrote up an example to guide you in the direction of classifying by relying on text instead of structural similarity between the documents.

My folder is organized like this:

I have two training images and three test images. The training images are in the template folder and the images I want to classify are in the toBeClassified folder. Here are my input images:

Now that we got that out of the way, here's a function that can get text from the images:

def text_from_image(filename):
    '''
    Load image as grayscale by using cv2.irmead(_, 0)
    extract text using pytesseract, we use the text for classification
    '''
    imGray = cv2.imread(filename, 0)
    text = pytesseract.image_to_string(imGray)
    return text

Now that we have the text, we can use it together with some labels that we put as training reponses to train the model:

texts = [text_from_image("Templates/"+filename) for filename in os.listdir("Templates/")] # get the texts
labels = ["Shipping", "Invoice"] # label the texts

These lines are used to get the model and to train it on the templates:

model = make_pipeline(TfidfVectorizer(), MultinomialNB()) # init the model
model.fit(texts, labels) # fit the model

Now we can go with two approaches, either we are just satisfied with the class with the highest probability or we go with predicting the probabilities.

The first approach:

def classify_with_model(filename, model):
    '''
    We pass the filename of the images to be classified and the model to get the 
    class with the highest probability, not very good but good for now
    '''
    text = text_from_image(filename)
    category = model.predict([text])[0]
    return category

If we loop through the test images, we get:

for filename in os.listdir("toBeClassified/"): # loop through the test images
    category = classify_with_model("toBeClassified/"+filename, model) # get the class with the highest prob
    print(f'Document {filename} is classified as {category}') # print out results

# Document Bill1.png is classified as Invoice
# Document Shipping1.jpg is classified as Shipping
# Document ShippingInvoice1.png is classified as Invoice

Notice the last image, is kind of a hybrid that I found online. So looking at the probabilities is quite essential in some cases. As the joke when it comes to SVM learners: A healthy person goes to the doctor for a cancer screening. The doctor uses a state-of-the-art SVM machine learning model with 100% accuracy in identifying different types of cancer. After the test, the doctor comes back and says, "Good news and bad news. The good news: our model is perfect at diagnosing every type of cancer. The bad news: it doesn't know how to say you're healthy."

---

In any case, no time for humour, or my attempt at it. We go with probabilities:

def classify_with_probability(filename, model):
    '''
    This is to classify with probability, a bit better to see the confidences
    '''
    text = text_from_image(filename)
    probabilities = model.predict_proba([text])[0]  # get probabilities for each class
    categories = model.classes_  # get the class labels
    bestMatchIdx = probabilities.argmax()  # index of the highest probability
    bestMatch = categories[bestMatchIdx]  # class with the highest probability
    confidence = probabilities[bestMatchIdx]  # probability of the best match
    return bestMatch, dict(zip(categories, probabilities)) # return everythin

And the results are:

for filename in os.listdir("toBeClassified/"):
    category = classify_with_probability("toBeClassified/"+filename, model)
    print(f'Document {filename} is classified as {category[0]} with the following confidence:')
    print(category[1])
    print("_______________________")
#Document Bill1.png is classified as Invoice with the following confidence:
#{'Invoice': 0.5693790615221315, 'Shipping': 0.4306209384778673}
#_______________________
#Document Shipping1.jpg is classified as Shipping with the following confidence:
#{'Invoice': 0.38458825025403914, 'Shipping': 0.6154117497459587}
#_______________________
#Document ShippingInvoice1.png is classified as Invoice with the following confidence:
#{'Invoice': 0.5519774748181495, 'Shipping': 0.4480225251818504}
#_______________________

Here, we see the probabilities as well, this can be helpful if you want to classify your images into multiple clases at the same time, e.g., invoice and shipping form.

I hope this helps you in some way, as I said, and as Christoph mentioned, going with features on those documents will be a wild attempt.

The imports:

import cv2
import pytesseract
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
import os
pytesseract.pytesseract.tesseract_cmd = "C:/Program Files/Tesseract-OCR/tesseract.exe"