OpenCV FFMPEG RTSP Camera Feed Errors

Question

I'm getting these errors at random times when saving frames from an rtsp camera feed. The errors happen at different times, usually after 100-200 images have been saved, and the errors themselves are not always exactly the same. They cause the images that are saved at the time of the error to be distorted either to the point of being completely grey or contain distorted pixels.

#Frame_142 - [hevc @ 0c3bf800] The cu_qp_delta 29 is outside the valid range [-26, 25].

#Frame_406 - [hevc @ 0b6bdb80] Could not find ref with POC 41

I've tried implementing the code in both python and c++ with the same result. Also tried saving as .png instead of .jpg. The rtsp feed works fine when using imshow to display the camera, the problem only appears to happen when trying to save the frames. From what I can gather the errors have to do with ffmpeg but google isn't much help for these types of errors.

#include <iostream>
#include <opencv2\opencv.hpp>
#include <chrono>
#include <thread>

using namespace std;
using namespace cv;

int main() {

    VideoCapture cap("rtsp://admin:admin@192.168.88.97/media/video1");
    if (!cap.isOpened())
        return -1;

    for (int i = 0; i < 500; i++)
    {
        Mat frame;
        cap >> frame;
        imwrite("C:\\Users\\Documents\\Dev\\c++\\OpenCVExample\\frames\\frame" + std::to_string(i) + ".png", frame);
        cout << i << "\n";
        std::this_thread::sleep_for(std::chrono::milliseconds(10));

    }

    return 0;
}

Answer 1

OpenCV and RTSP streams are a little tricky. This usually happens when the frames of the stream cannot be read fast enough. In that case (somewhere) a buffer receiving RTSP frames will be filled faster than is can be emptied; when the buffer is full it will be flushed and will restart filling from the most recent frame. This will corrupt the stream decoding until the next keyframe.

My suggestion is to remove the sleep even if 10ms is pretty small. Since this won't be enough (:P), the only solution is to implement a thread constantly reading frames from the cv::VideoCapture and discarding the ones grabbed while the main thread is busy saving the previous frame.

A simple implementation could be:

#include <opencv2/opencv.hpp>
#include <thread>

class VideoSourceHandler
{
public:
  // Video file or IP camera
  VideoSourceHandler( const std::string & source ) :
    mCapture( source ),
    mRun( false )
  {
  }

  // USB camera
  VideoSourceHandler( int webcamID ) :
    mCapture( webcamID ),
    mRun( false )
  {
  }

  // start and stopCapture can be squashed into C'tor and D'tor if you want RTTI
  void startCapture()
  {
    mRun = true;
    mLoopThread = std::thread( &VideoSourceHandler::threadLoop, this );
  }

  void stopCapture()
  {
    mRun = false;
    mLoopThread.join();
  }

  cv::Mat getFrame()
  {
    std::this_thread::yield(); // Be nice
    const std::lock_guard<std::mutex> lock( mMutex );
    return mCurrentFrame;
  }

private:
  void threadLoop()
  {
    while( mRun )
    {
      // Sleep if you want to "control" FPS
      {
        const std::lock_guard<std::mutex> lock( mMutex );
        mCapture >> mCurrentFrame;
      }
      std::this_thread::yield(); // Be nice
    }
  }

  cv::VideoCapture mCapture;
  std::thread      mLoopThread;
  std::mutex       mMutex;
  bool             mRun;
  cv::Mat          mCurrentFrame;
};

int main()
{
  VideoSourceHandler vsh( 0 );
  vsh.startCapture();
  while( true )
  {
    cv::Mat frame = vsh.getFrame();
    if( frame.empty() )
      continue;

    cv::imshow( "Test", frame );
    char key = cv::waitKey( 1 );
    if( key == 'q' || key == 27 )
      break;

  }
  vsh.stopCapture();
  return 0;
}