Get the coordinates of the objects found by a template

Question

I wanted to make a bot for a game that looks for a certain item on the floor, and then clicks on it. I managed to get the first part right (it even draws a rectangle around it) but whats embarrassing is that i cant get the coordinates of that object right. I use cv2.matchTemplate method. This is my code:

import numpy as np
import pyautogui

img_bgr =  cv2.imread('gra.png')
img_gray = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2GRAY)

template = cv2.imread('bones2.png', 0)

w, h = template.shape[:: -1]

res = cv2.matchTemplate(img_gray, template, cv2.TM_CCOEFF_NORMED)
threshhold = 0.90
loc = np.where( res >= threshhold)
for pt in zip(*loc[:: -1]):
    cv2.rectangle(img_bgr, pt, (pt[0] + w, pt[1] + h),(0, 255, 255), 2 )
    #here i wanted to move the mouse to the coordinates of a found item, however
    #i cant get these two right ↓        ↓
    pyautogui.moveTo(           ?     ,  ? ,duration=0.5)

cv2.imshow('znalezione', img_bgr)

cv2.waitKey()
cv2.destroyAllWindows()

I tried this:

 pyautogui.moveTo( (pt[0] *2  + w)/2  , (pt[1] *2 + h)/2 ,duration=0.5)

but this doesn't work at all. Can someone explain to me what pt is at all and how to get the coordinates?

Also here is a screen shot of what i achieved so far:

Answer 1

From my understanding, both OpenCV and pyautogui uses the same coordinate system as illustrated with an example 1920x1080 resolution.

0,0       X increases -->
+---------------------------+
|                           | Y increases
|                           |     |
|   1920 x 1080 screen      |     |
|                           |     V
|                           |
|                           |
+---------------------------+ 1919, 1079

OpenCV's cv2.rectangle function takes the top left coordinates and the bottom right coordinates of the rectangle as parameters. Since you were able to draw the bounding box in your image, you have the correct coordinates of the ROI you want to examine. From the docs the moveTo function takes two parameters: x and y. Assuming you want to move the mouse to the center of the bounding box, you can do

pyautogui.moveTo(pt[0] + w/2, pt[1] + h/2, duration=0.5)