python, how do i matchtemplate transparent image with opencv

Question

im using matchtemplate to detect 67x45 sqaure on background. i think my code works fine without any problem but the problem is i have to set high threshold for it to detect successfully otherwise it would give so many false detections. so i tried changing method to cv2.TM_CCOEFF_NORMED but it didnt work well. i also tried to look for opencv document to understand how all these methods work but its really hard for me to understand since they just provide math formulas.. so my question is am i fine with my code? or is there a better way to accomplish what i want to do? (im also not sure if im using 'template' parameter in matchtemplate in the correct way)

import cv2
import numpy as np 
import win32gui, win32ui, win32con

    
def imagesearch(per):


    filename = 'target.png'

    img = cv2.imread(filename, cv2.IMREAD_GRAYSCALE)
    img1 = cv2.imread(filename)

    template = cv2.imread('./map/6745.png', cv2.IMREAD_GRAYSCALE)

    w, h = template.shape[::-1]

    meth = [cv2.TM_CCOEFF, cv2.TM_CCOEFF_NORMED, cv2.TM_CCORR, cv2.TM_CCORR_NORMED, cv2.TM_SQDIFF, cv2.TM_SQDIFF_NORMED]

    res = cv2.matchTemplate(img, template, meth[3], None, template)


    threshold = per 
    loc = np.where(res>=threshold)


    if loc[0].any():



        for pt in zip(*loc[::-1]):
            cv2.rectangle(img1, pt, (pt[0] + w, pt[1] + h), (0,0,255), 1) 


    cv2.imshow("dst", img1)
    cv2.waitKey(0)
    cv2.destroyAllWindows()


imagesearch(0.99)

image

template

result with threshold = 0.99

result with threshold = 0.95

as @Christoph Rackwitz suggested,

import cv2
import numpy as np 
import win32gui, win32ui, win32con


def imagesearch():


    filename = 'target.png'

    img = cv2.imread(filename, cv2.IMREAD_GRAYSCALE)
    img1 = cv2.imread(filename)

    template = cv2.imread('./map/6745.png', cv2.IMREAD_GRAYSCALE)

    w, h = template.shape[::-1]

    meth = [cv2.TM_CCOEFF, cv2.TM_CCOEFF_NORMED, cv2.TM_CCORR, cv2.TM_CCORR_NORMED, cv2.TM_SQDIFF, cv2.TM_SQDIFF_NORMED]

    method = meth[5]
    res = cv2.matchTemplate(img, template, meth[5], None, template)

    if method in [cv2.TM_SQDIFF, cv2.TM_SQDIFF_NORMED]:

        min_val,max_val,min_loc, max_loc = cv2.minMaxLoc(res)
        top_left = min_loc
         
        bottom_right = (top_left[0]+w,top_left[1]+h)
        cv2.rectangle(img1,top_left,bottom_right,255,1)

    cv2.imshow("dst", img1)
    cv2.waitKey(0)
    cv2.destroyAllWindows()


imagesearch()

this one doesnt work either.

Answer 1

I am using matchtemplate to detect 67x45 square on background.

In the special case of a bounding-box-like rectangle it makes sense to detect it using own method working directly on the numpy image array instead of using OpenCV matching functions with a template image and coping with different matching algorithms and thresholds.

The code below detects any number of rectangles with one-pixel wide sides of the shape given by the template image. It works for any rectangle line color so it doesn't matter if it is a white rectangle on dark background or black rectangle on bright background or if the grey value of the rectangle equals that of the background because it uses full color information.

To simplify the search conditions the BGR-tuples of the image are converted to single integer values representing RGB-colors so the image array for the search algorithm is a pure 2D one (like it is when using grayscale).

To make sure that rectangles within a single-colored background are not detected one inner pixel in the left topmost corner of a rectangle is checked and has to have another color as the rectangle. This condition could lead to a not detected rectangle and require then checking more than one pixel, but extending the check to more than one pixel can be easily implemented into the sequence of checked conditions if it turns out it will be necessary. And if the background image does not have large areas of single color of the size of the rectangle the pixel check can be entirely removed from the sequence of checked conditions.

The Python loops in the provided code can maybe be replaced by a vectorized numpy approach, so comments on how to accomplish that are welcome.

import cv2
import numpy as np 

img_filename = "openCV_squareOnBackground.png"
ibgr = cv2.imread(img_filename, cv2.IMREAD_UNCHANGED)
assert ibgr.shape[2] == 3
sizeY, sizeX, _ = ibgr.shape 
# Convert uint8 BGR tuples to uint32 value: 
iint32 = np.dstack((ibgr, np.zeros(ibgr.shape[:2], 'uint8'))).view('uint32').squeeze(-1)

img_filename = "openCV_squareOnBackground_template.png"
grey_t = cv2.imread(img_filename, cv2.IMREAD_GRAYSCALE)
t_sizeY, t_sizeX = grey_t.shape

rectAt = []
for x in range(sizeX - t_sizeX):
    for y in range(sizeY - t_sizeY):
        if \
           (iint32[y,x] == iint32[y+1:y+t_sizeY  ,x              ]).all() and \
           (iint32[y,x] == iint32[y              ,x+1:x+t_sizeX  ]).all() and \
           (iint32[y,x] == iint32[y+1:y+t_sizeY  ,x    +t_sizeX-1]).all() and \
           (iint32[y,x] == iint32[y    +t_sizeY-1,x+1:x+t_sizeX  ]).all() and \
           (iint32[y,x] != iint32[y+1,x+1]) and \
                       True:
            rectAt.append((x,y))

print(rectAt)

for x, y in rectAt:
    cv2.rectangle(ibgr, (x-1,y-1), (x+t_sizeX, y+t_sizeY), (0,255,255), 1) 

cv2.imshow("ibgr", ibgr)
cv2.waitKey(0)
cv2.destroyAllWindows()

The code finds in the provided image exactly one rectangle:

at [(89, 13)] .