How to move my crosshair to the bounding box my model creates

Question

I am working on a project involving creating an AI-powered bot to full play Counter-Strike by itself. I have a working script that uses the YOLOv8 object detection algorithm to identify players on the screen, and draw a the bounding box around them.

I'm encountering an issue where the bot will consistently look downward when an enemy is detected on the screen. I have tried adjusting the mouse movement calculations and implementing crosshair adjustments, but the issue persists. I even looked at a similar implementation using win32, when I tried implementing that code to my project it still would have the bot look as far down as it could. Has anyone encountered a similar behavior or have suggestions on what might be causing this? I would greatly appreciate any insights, advice, or strategies that could help me pinpoint and address the cause of the issue.

Thank You!

Here is my code below:

import os
import time
from ultralytics import YOLO
import cv2
import numpy as np
from windowcap import WindowCapture
import win32gui, win32ui, win32con, win32api
import pydirectinput


wincap = WindowCapture('Counter-Strike: Global Offensive - Direct3D 9')
model_path = os.path.join('.','runs','detect','train10','weights','last.pt')
model = YOLO(model_path)

while True:

    frame = wincap.get_screenshot()

    H, W, _ = frame.shape 
    threshold = 0.5


    results = model(frame)[0]

    for result in results.boxes.data.tolist():
        x1, y1, x2, y2, score, class_id = result

        if score > threshold:
            cv2.rectangle(frame, (int(x1), int(y1)), (int(x2), int(y2)), (0, 255, 0), 4)
            cv2.putText(frame, results.names[int(class_id)].upper(), (int(x1), int(y1 - 10)),
                        cv2.FONT_HERSHEY_SIMPLEX, 1.3, (0, 255, 0), 3, cv2.LINE_AA)

            box_center_x = (x1 + x2) / 2
            box_center_y = (y1 + y2) / 2
            screen_width = win32api.GetSystemMetrics(0)
            screen_height = win32api.GetSystemMetrics(1)

            # moving crosshair to model box
            target_x = int(screen_width * box_center_x / W)
            target_y = int(screen_height * box_center_y / H)
            pydirectinput.moveTo(target_x, target_y)
    
    cv2.imshow("COMPUTER VISION",frame)

    if cv2.waitKey(1) == ord('q'):
        cv2.destroyAllWindows()

Answer 1

The box center coordinates from xyxy box format:

box_center_x = x1 + (x2 - x1) / 2
box_center_y = y1 + (y2 - y1) / 2

Or you can get them directly from the results object. For the single detection it will be:

x_centre, y_centre, w, h = results[0].boxes[0].xywh.tolist()