Search code examples
c++opencvgstreamerrtsp

Displaying RTSP stream with OpenCV and gstreamer

I have purchased an IP camera which I am trying to connect to using RTSP. The RTSP connection URL is rtsp://admin:@192.168.0.27/channel=1&stream=0.554. I am using OpenCV to open and display the stream:

#include <opencv2/opencv.hpp>

int main() {
   cv::VideoCapture cap;
   if (!cap.open("rtsp://admin:@192.168.0.27/channel=1&stream=0.554")) {
        std::cout << "Unable to open video capture\n";
        return -1;
    }

    while(true) {
        cv::Mat frame;

        auto ret = cap.grab();
        cap >> frame;

        if (frame.empty()) {
            break; // End of video stream
        }

        cv::resize(frame, frame, cv::Size(640, 480));

        // Do other stuff here with frame

        cv::imshow("frame", frame);

        if (cv::waitKey(10) == 27) {
            break; // stop capturing by pressing ESC
        }
    }

    return 0;
}

When I run the program, it will connect successfully and display a few frames, but then will start to lag badly and usually hang and output some error like this before crashing:

[h264 @ 0x558ae8e601a0] error while decoding MB 93 40, bytestream -11

I am not sure why I am having issues displaying the stream consistently. Additionally, when it is able to display the stream, I find that it is quickly become out of sync (note I am doing some heavy processing on the frame which takes quite a lot of time). As in, it is not displaying the real time frame, but there is a growing lag.

How can I also ensure to use the "latest" frame, and discard all the other ones which may have accumulated in some buffer. Also, any ideas why it is crashing and how I can improve the streaming?

I was able to find this SO post which deals with getting the latest frame using gstreamer. When I modify my video capture string to utilize gstreamer, it works a bit better.

Here is the modified connection string: "rtspsrc location=rtsp://admin:@192.168.0.27/channel=1&stream=0.554 ! decodebin ! videoconvert ! appsink max-buffers=1 drop=true"

I have no experience with gstreamer so I am not sure what it is doing, but it seems to improve things. However, once it a while, it will all go grey and only display the pixels when there is movement, as shown in the following images. With my experince with codecs, I believe the reference frame is missing, but I am not really sure. Any ideas on how to fix this? If I am not using the correct gstreamer parameters, please make a suggestion as to what I should be using for fast streaming (always using the latest frame). As I mentioned, I have minimal experience with gstreamer. Thanks for the help!

enter image description hereenter image description here

Solution

  • This may be due to packet loss of the network transmission. You can give it a try to modify the URL to use the rtspt:// protocol. This will try to establish a TCP transmission which should prevent packet loss on your receiving side.