Esp32-Cam stream in OpenCV+YOLOv8 is so laggy

Question

I'm using ESP32-Cam WebServer and capture the cam stream in python OpenCV+YOLOv8.
It works, but there is significant lag.
The delay is approximately 2 seconds after what the camera captures.

Here's my code:

import cv2
from ultralytics import YOLO

model = YOLO('yolov8n.pt')
cap = cv2.VideoCapture("http://ip")


while cap.isOpened():
    success,frame = cap.read()


    if success:
        results = model(frame)

        annotated_frame = results[0].plot()

        cv2.imshow("YOLOv8 Inference",annotated_frame)


        if cv2.waitKey(1) & 0xFF ==ord("q"):
            break
    else:
        break
    
cap.release()
cv2.destroyAllWindows()

Is this due to insufficient hardware performance of the ESP32 or are there any modifications that can be made to the code?

Answer 1

The AI processing slows everything down.

This causes video buffers to inflate, which causes apparent latency.

Drop the AI processing as a test. You'll probably see it play back with less delay.

To fix this, run the AI processing in its own thread, and make sure you don't force it to process every frame.