Find image in scaled Image

Question

I have to find an image during a stream of desktop. My code work, but if the image, during the stream, is resized, the program not work. How can I solve this problems?

from PIL import ImageGrab
import numpy as np
import cv2

template = cv2.imread('piccola.png') #image to find
w, h = template.shape[:-1]

while 1:
    img = ImageGrab.grab(bbox=(0,0,800,600)) #bbox specifies specific region (bbox= x,y,width,height *starts top-left)
    img_np = np.array(img) #this is the array obtained from conversion
    #frame = cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)

    res = cv2.matchTemplate(img_np, template, cv2.TM_CCOEFF_NORMED)
    threshold = .85
    loc = np.where(res >= threshold)
    for pt in zip(*loc[::-1]):  # Switch columns and rows
        cv2.rectangle(img_np, pt, (pt[0] + h, pt[1] + w), (0, 0, 255), 2)

    cv2.imshow("output", img_np)
    if cv2.waitKey(25) & 0xFF == ord('q'):
       cv2.destroyAllWindows()
       break

Answer 1

Instead of using cv2.matchTemplate, you can actually extract features from your template image, i.e. extract features such as SIFT/ORB/KAZE/BRISK and match them against by extracting the same features from the grabbed image. You can set up a threshold for the matching criteria.

you can read more about feature description and matching here - https://docs.opencv.org/3.4/d5/dde/tutorial_feature_description.html

Sample code for your understanding.

import cv2
import numpy as np
img1 = cv2.imread("template.jpg", cv2.IMREAD_GRAYSCALE)
img2 = cv2.imread("image.jpg", cv2.IMREAD_GRAYSCALE)
# ORB Detector
orb = cv2.ORB_create()
kp1, des1 = orb.detectAndCompute(img1, None)
kp2, des2 = orb.detectAndCompute(img2, None)
# Brute Force Matching
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
matches = bf.match(des1, des2)
matches = sorted(matches, key = lambda x:x.distance)
#drawing the matches
matching_result = cv2.drawMatches(img1, kp1, img2, kp2, matches[:50], None, flags=2)

you can filter the matches which have the distance > 0.7 (usual threshold) and check the percentage of matches. Based on that you can decide how well it's finding the similar images.

Methods like SIFT is patented but performs well.

Methods like ORB is fastest, but not invariant to scale.

you can try methods like KAZE and AKAZE.