Search for an icon on an image OpenCV

Question

I am new to the field of computer vision and have a challenge finding an fish icon within an image(example1, example2, example3). Specifically, I need to locate the fish icon from a game and obtain its coordinates. The icon's size changes depending on its position within the water, as well as its color when the fish is being pulled. Additionally, the effects within the game often hinder the process of searching for the icon. The main issue is that the icon frequently changes its scale. As a result, the methods I have attempted to use for searching have gotten lost or have only rarely yielded a normal result.

To begin with, I tried using Template matching, but it doesn't look good if the icon is a different size. I also tried to make an array of several sizes, but it worked very unstable and slowed down the algorithm very much.

std::vector<Mat> fishIcon;
const std::vector<double> SCALES = { 0.4, 0.5, 0.6, 0.8, 1.0, 1.2, 1.4 };

//Code for creating multiple masterstabs
for (double scale : SCALES) {
    Mat resIcon;
    resize(sourceIcon, resIcon, Size(0, 0), scale, scale);

    fishIcon.push_back(resIcon);
}

//A method for searching for an icon in an image
bool FindObject(Mat& templateForFind, Mat& screenshot, Point& maxLoc, Mat& result, bool debug, double confid) {
    matchTemplate(screenshot, templateForFind, result, TM_CCOEFF_NORMED);
    double minVal, maxVal;
    minMaxLoc(result, &minVal, &maxVal, NULL, &maxLoc);

    if (debug) {
        std::cout << maxVal << std::endl;
    }

    if (maxVal >= confid) {
        return true;
    }
    else {
        return false;
    }
}

//Going through all the scales until you find the right one
for (Mat ic : fishIcon) {
    if (FindObject(ic, screenshotGray, max_loc, result, true, 0.8)) {
        fishIconPosition = Point(max_loc.x, max_loc.y);
        found = true;
        break;
    }
}

And I tried the basic search method using ORB and histogram, which were shown on the OpenCV website, but they did not show good results either.

Therefore, I want to find out if there are no simpler and better methods? I will be grateful for any hint

Answer 1

I tried to obtain the icon using some image processing. I used the 3 images you provided and the below solution in python works. The assumption is that the icon border is white as in sample images. Explanation of each step is provided in comments. Hope this helps.

import cv2
import numpy as np

fish_images = ["fish1.png","fish2.png","fish3.png"] #Image name list to process

#For loop to process
for img in fish_images:
    #Read image
    image_org = cv2.imread(img)
    #Convert to grayscale
    grayscale_image = cv2.cvtColor(image_org, cv2.COLOR_BGR2GRAY)
    #Use threshold to get the white part in images , this will select the icon part also0 as it's white borders.
    ret, thresholded_image = cv2.threshold(grayscale_image, 200, 255, cv2.THRESH_BINARY)

    #Apply some morphological operatipn sto remove noise
    kernel = np.ones((5, 5), np.uint8)
    thresholded_image = cv2.morphologyEx(thresholded_image, cv2.MORPH_OPEN, kernel)
    thresholded_image = cv2.morphologyEx(thresholded_image, cv2.MORPH_CLOSE, kernel)

    #Dinf contours in the image
    contours, hierarchy = cv2.findContours(thresholded_image, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
    contour_image = np.zeros_like(thresholded_image)

    #This loops go throught the contours and find the ones with area >700 and <1500
    # This is found through trial. You can adjust this accoring to your use
    #The I find the are of contour and bounding rectangle of the contour.
    # For rectangle, the contours the area and rectangular area difference will be the lowest
    selected_ctr = None
    selected_bounding_rect = []
    lowest_area_diff = 100000
    for ctr in contours:
        ctr_area = cv2.contourArea(ctr)
        if ctr_area > 700 and ctr_area < 1500:  # ADJUST THIS ACCORDING TO YOUR NEED.
            bounding_rect = cv2.boundingRect(ctr)
            bounding_rect_area = bounding_rect[2]*bounding_rect[3]
            area_difference = abs(ctr_area - bounding_rect_area)
            if area_difference < lowest_area_diff: #Compare area difference select one with lowest difference to avoid unnecessary areas
                selected_ctr = ctr
                lowest_area_diff = area_difference

    #The selected contour is used to create the mask
    #Using the mask the icon is obtained from image
    if selected_ctr is not None:
        cv2.drawContours(contour_image, [selected_ctr], -1, 255, thickness=-1)
        selected_image = cv2.bitwise_and(image_org, image_org, mask=contour_image)
        #Save result image
        cv2.imwrite(img+"_result.png",selected_image)

Sample output :