Opencv/Python - how can I get the coordinates of detected areas after image processing

Question

My goal is to write a bot playing minesweeper, but I get stuck at the point, when I want to tell the bot, where the squares are. I tried a lot of different functions. First of all, my tool grabs a screenshot of a predefined area. This picture looks like this:screenshot of game board

after that I want to fill a numpy array in this way:

info_map = np.ones((board_height=9, board_width=9),
                   dtype = np.uint8)*11
>array([[11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11],
        [11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11],
        ...
        [11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11]], dtype=uint8

11 stands in this example for "undiscovered" or a blue, untouched square.

with the help of the opencv square detector from the examples I get a huge list with detected points (nearly 5000points for a 9x9 grid). I don't know how to remove all the noise, to get the right coords of the corners. so I came to the next example showing below:

My last try was the following:

import glob
import cv2
import numpy as np
import sys   

def canny_edge():
"""This function is taken from SentDex from one of his fantastic Python
tutorials.
https://pythonprogramming.net/canny-edge-detection-gradients-python-opencv-tutorial/
"""
for fn in glob('full_snap.png'):
    img = cv2.imread(fn)
    while(1):
        hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
        lower_red = np.array([30,150,50])
        upper_red = np.array([255,255,180])
        mask = cv2.inRange(hsv, lower_red, upper_red)
        #res = cv2.bitwise_and(img, img, mask= mask)
        laplacian = cv2.Laplacian(img, cv2.CV_64F)
        #edges = cv2.Canny(img,100,200)

        cv2.imwrite(os.getcwd() + '\\laplace.png', laplacian)

This is the saved image: laplace.png

here I tried to do a for loop over the Laplacian to check if the pixel is blue or not, and take the first one as an anchor to tell the bot, that from the anchor (starting point) there is a square every x pixel in the x direction and in the y. But is there a better way to do it automatically?

but my main question is how to connect the image processed squares into an numpy array, that I can tell the bot click_square(1,2) and he knows at which pixels is the square from row 1 and column 2.

Hint: My coordinate system starts in the upper left corner of the screen.

Thank you for your help.

Answer 1

I don't know what minesweeper is but from what i understood, you want to know which tiles are blue and then make necessary changes to your numpy array (correct me if its something else, i'll edit or remove the answer).

Here's the output after making four tiles blue:

What i did ?

First i thresholded the blue colour, found contours in the range approximately as that of your tiles.

Found their centers to see what pattern they follow -

They were all separated by some 55-57 pixels (both x,y coordinates). Rest is simple.

for i in range(len(coords)):
np_arr[int(coords[i][1]/57)][int(coords[i][0]/57)]=0

The blue tile coordinates are stored in coords, np_arr is my array.