Getting screen pixels on mouse press

Question

I would like to obtain the screen pixels at the mouse coordinates when I click anywhere on the screen (including outside my application).

As a first step, I take a screenshot as follows:

from PyQt5 import QtWidgets, QtGui

class MainWindow(QtWidgets.QMainWindow):
    def __init__(self):
        super(MainWindow, self).__init__()

        w = QtWidgets.QWidget()
        layout = QtWidgets.QVBoxLayout()
        w.setLayout(layout)
        self.setCentralWidget(w)

        screenshot = QtWidgets.QApplication.primaryScreen().grabWindow(w.winId())

How can I map the coordinates where I have clicked to the screenshot?

I think the next step would be to convert the screenshot to a Pixmap to extract the pixels but I have no idea how to connect this with the mouse press coordinates.

Answer 1

The only way to capture click events outside any Qt window is through grabMouse(). Consider that this only works on Linux:

Note: On Windows, grabMouse() only works when the mouse is inside a window owned by the process. On macOS, grabMouse() only works when the mouse is inside the frame of that widget.

Also note that grabbing mouse might be dangerous:

Warning: Bugs in mouse-grabbing applications very often lock the terminal. Use this function with extreme caution, and consider using the -nograb command line option while debugging.

As soon as the widget has grabbed the mouse, it will capture all mouse events (including mouse wheel), no matter their position, so it's also very important to release it as soon as possible. Then, there is no need to map the coordinates: the mousePressEvent already provides global positions based on screen coordinates, and since you already took a screenshot of the whole screen area, those coordinates will match.

Finally, to get the pixel data, convert the QPixmap provided by grabWindow, convert it to a QImage and use pixelColor.

Because of the platform limitations of grabMouse, in the following example I'm creating a transparent window that acts as a "fake" mouse grabber.

class MainWindow(QtWidgets.QMainWindow):
    def __init__(self):
        super(MainWindow, self).__init__()

        w = QtWidgets.QWidget()
        layout = QtWidgets.QVBoxLayout()
        w.setLayout(layout)
        self.setCentralWidget(w)
        self.grabButton = QtWidgets.QPushButton('Grab')
        layout.addWidget(self.grabButton)
        self.label = QtWidgets.QLabel('Waiting')
        layout.addWidget(self.label)
        self.grabButton.setCheckable(True)
        self.grabButton.toggled.connect(self.startGrab)

        self.desktopId = QtWidgets.QApplication.desktop().winId()

        if sys.platform != 'linux':
            self.grabber = QtWidgets.QWidget(flags=QtCore.Qt.FramelessWindowHint)
            self.grabber.setAttribute(QtCore.Qt.WA_TranslucentBackground)
            self.grabber.installEventFilter(self)
        else:
            self.grabber = None

    def grabPixel(self, event):
        buttonRect = self.grabButton.rect()
        buttonRect.translate(self.grabButton.mapToGlobal(QtCore.QPoint()))
        if event.globalPos() not in buttonRect:
            screenshot = QtWidgets.QApplication.primaryScreen().grabWindow(self.desktopId).toImage()
            pixel = screenshot.pixelColor(event.globalPos())
            self.label.setText('Click on {}x{}\nRGB: {}'.format(event.globalX(), event.globalY(), pixel.name()))
        self.grabButton.toggle()

    def startGrab(self, grab):
        if grab:
            if self.grabber:
                rect = QtCore.QRect()
                for screen in QtWidgets.QApplication.screens():
                    rect |= screen.geometry()
                self.grabber.setGeometry(rect)
                self.grabber.show()
            else:
                self.grabMouse()
        else:
            self.releaseMouse()

    def eventFilter(self, source, event):
        if source == self.grabber and event.type() == QtCore.QEvent.MouseButtonPress:
            self.grabPixel(event)
            self.grabber.hide()
        return super().eventFilter(source, event)

    def mousePressEvent(self, event):
        self.grabPixel(event)