Search code examples
pythonnumpyqtpyqt5

How to crop a live video feed in PyQt5? QImage looks sheared

I have a PyQt5 GUI application. I read a frame from the camera to select a smaller area on it, and later I would like to display only that area. I correctly select the area, and store it in a QRect. But when I try to crop the frames to display and to process later, the displayed live video just sometimes works correctly. but most of the times it looks like on this picture. It becomes slanted, the right corner should be the left corner. And the pink circle should be in the middle of the video, so it is kinda wrong.

Image example

I have this Worker1 class, where I process the displayed frames.

class Worker1(QThread):
    ImageUpdate = pyqtSignal(QImage)

    def __init__(self, canvas, selection):
        super().__init__()
        self.ThreadActive = True
        self.canvas = canvas
        self.anim = None
        self.selection = selection

    def run(self):
        Capture = cv.VideoCapture(0)

        old_closest_contour = None
        d_tresh = 300
        P_color = (0, 255, 0)
        global i
        i = 0

        while self.ThreadActive:
            i += 1
            ret, frame = Capture.read()
            if ret:
                if self.selection is None or self.selection.isEmpty():
                    cropped_frame = frame
                else:
                    a = int(self.selection.x())
                    b = int(self.selection.y())
                    width = int(self.selection.width())
                    height = int(self.selection.height())
                    #print(str(a) + " " + str(b) + " " + str(width) + " " + str(height))
                    cropped_frame = frame[b: b + height, a: a + width].copy()
                    #print(cropped_frame.shape[1])
                    #print(cropped_frame.shape[0])

                ### other processing lines, those are not relevant in the displaying ###

                Image = cv.cvtColor(cropped_frame, cv.COLOR_BGR2RGB)
                img = cv.medianBlur(Image, 25)

                cv.circle(img, (x, y), r, P_color, 1)
                cv.circle(img, (x, y), 4, P_color, -1)
                cv.circle(img, point, 0, (255, 0, 255), 10)

                ConvertToQtFormat = QImage(img.data, img.shape[1], img.shape[0], QImage.Format_RGB888)
                self.ImageUpdate.emit(ConvertToQtFormat)

        Capture.release()
Solution

  • We have to set bytesPerLine argument.
    Replace ConvertToQtFormat = QImage(img.data, img.shape[1], img.shape[0], QImage.Format_RGB888) with:

    ConvertToQtFormat = QImage(img.data, img.shape[1], img.shape[0], img.strides[0], QImage.Format_RGB888)

    img.strides[0] applies the number of bytes in each line of img.
    When img.shape[1] = 100, img.strides[0] is usually equal 100*3 = 300 bytes when there are 3 bytes per pixel.

    The reason that we have the strides is that there are cases when lines are not continuous in memory, and the stride doesn't equal width * 3.

    The equivalent parameter in QImage object is bytesPerLine.
    By default, QImage object assumes that bytesPerLine is a multiple of 4.
    In case width*3 is not a multiple of 4, padding bytes are assumed to be present at the end of each line.
    For example, when width = 101, bytesPerLine is 304 instead of 303 (1 padding byte is assumed).
    (The multiple of 4 assumption is originated from the BMP image format).

    In our case:

    • When img.shape[1] is a multiple of 4, the image is going to look correct.
    • When img.shape[1] is a not multiple of 4, the image is going to be "slanted".
      In our case the NumPy array has no padding bytes, and there is a mismatch between Qt bytesPerLine and NumPy strides.

    The solution is setting the bytesPerLine parameter (using an overloaded QImage constructor).
    In general, when converting from NumPy to QImage, we should set img.strides[0] as bytesPerLine argument.