How to get screen coordinates from color detection resultant area obtained from openCV with python?

Question

I am trying to work with a pen having green cap tip to navigate mouse cursor using webcam but how can I get the coordinates of cap image on screen so that I can give it as input to pyinput library move function.

Thanks in advance.

# Python program for Detection of a
# specific color(green here) using OpenCV with Python

import cv2
import numpy as np
import time

cap = cv2.VideoCapture(0)


while (1):
    # Captures the live stream frame-by-frame

    _, frame = cap.read()
    hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
    lower_red = np.array([75, 50, 50])
    upper_red = np.array([100, 255, 255])
    mask = cv2.inRange(hsv, lower_red, upper_red)
    res = cv2.bitwise_and(frame, frame, mask=mask)
    cv2.imshow('frame', frame)
    cv2.imshow('mask', mask)
    cv2.imshow('res', res)

    **#code to output coordinates of green pen tip**

    k = cv2.waitKey(5) & 0xff
    if k == 27:
        break

cv2.destroyAllWindows()
cap.release()

Answer 1

You have all the things you need, just a couple of steps more are needed:

First find the non zero points in your mask, which should represent the tip.

points = cv2.findNonZero(mask)

Then you can averaged them to have a "unique point" which represents the tip.

avg = np.mean(points, axis=0)

Now, you can normalize this into a 0-1 value which can be later be used in any resolution... or you can normalize it directly to the resolution of the screen...

# assuming the resolutions of the image and screen are the following
resImage = [640, 480]
resScreen = [1920, 1080]

# points are in x,y coordinates
pointInScreen = ((resScreen[0] / resImage[0]) * avg[0], (resScreen[1] / resImage[1]) * avg[1] )

A few things to consider, first remember that opencv coordinate system origin is on the top left side pointing down and to the right.

 ---->
|
|   image
|
v

Depending on where and how you use these point coordinates, you may need to flip the axis....

Second, in OpenCV points are in x,y and (at least the resolutions I wrote manually) are in width and height... you may have to adapt the code to cope with this if needed :)

If you have any questions just leave a comment