Search code examples
opencvimage-processingedge-detection

Detect or split icons in a grid

I'm writing a third-party tool app for a smartphone game. In this game, there is a page showing icons in a grid:

Screenshot

I want to make a feature that can detect and split out these icons. I tried many things like Sudoku detection, Detecting grids in images. By following these tutorials, the best result I got is something like this:

enter image description here enter image description here

It's not a good result and I don't know what I can do next to process it. The reason why it's difficult I think is:

  • the background color is so strong.
  • the content within icons is much richer than the separator between them.
  • Actually, there is no separator line, but only frames being around icons.

So is there any algorithm that can detect square vaguely? Or any other approach/suggestion to solve this problem?

Solution

  • I suggest to consider the point that the borders colour of your icons is almost white, and the icons are contained in a grid.

    Hence, a proper thresholding will lead you into something like this:

    import cv2

BGR = cv2.imread('input.jpg')

Cpy = BGR.copy()

# Thresholding
Cpy[Cpy[...,0]!=Cpy[...,1]]=0
Cpy[Cpy[...,2]<200]=0

Cpy[Cpy>0]= 1

    th

    after that you need to find the corners:

    import numpy as np

rowSum = Cpy[...,0].sum(axis=0)
colSum = Cpy[...,0].sum(axis=1)

rows = np.zeros_like(Cpy)
cols = np.zeros_like(Cpy) 
mask = np.zeros_like(Cpy)

# Not sure if these values will work always
rows[:, rowSum>100] = 1
cols[colSum>200, :] = 1

mask = rows*cols

y0 = np.min(np.nonzero(mask.sum(axis=1))[0])
y1 = np.max(np.nonzero(mask.sum(axis=1))[0])

x0 = np.min(np.nonzero(mask.sum(axis=0))[0])
x1 = np.max(np.nonzero(mask.sum(axis=0))[0])

mask[y0:y1, x0:x1] = 1

mask1 = mask*rows
mask2 = mask*cols

mask = np.maximum(mask1, mask2)

    corners

    After that, you are free to use any method to detect your images as you have the corners of your icons, here I used Morphological dilation to process the corners, and labeling:

    SE = np.ones((16,16))
dilated = cv2.dilate(mask, SE)
dilated [...,1:3] = 0

from skimage.measure import label

labelled = label(1-dilated [...,0])

    mask

    Now you have your mask, thus you can detect your images:

    labelled[labelled==1] = 0
labelled[labelled >0] = 1

labelled = labelled.astype(np.uint8)

res = cv2.bitwise_and(BGR,BGR,mask = labelled)

cv2.namedWindow('Splitted Images', cv2.WINDOW_NORMAL)
cv2.imshow('Splitted Images', res)
cv2.waitKey(0)

    result