Search code examples
c++opencvnvidiavideo-processing

VideoReader in OpenCV is deprecated. Is there any update from OpenCV or any substitution?

I am doing video decoding to grab frame for some post video processing. I am using opencv Cuda so I really need video reader from opencv cuda. However, I realized:

In NVIDIA Video Codec SDK 8.2.15, it says that

Support for CUvideosource and the associated APIs including cuvidCreateVideoSource,cuvidCreateVideoSourceW, cuvidDestroyVideoSource, cuvidSetVideoSourceState, cuvidGetVideoSourceState, cuvidGetSourceVideoFormat, cuvidGetSourceAudioFormat will be removed from the decoder API in future SDK versions. Please note that the new decode sample applications in the SDK do not use these APIs, but use FFmpeg instead.

But to use "createVideoReader" in OpenCV Cuda, we need to link "dynlink_nvcuvid.h" that contains

tcuvidCreateVideoSource               *cuvidCreateVideoSource;
tcuvidCreateVideoSourceW              *cuvidCreateVideoSourceW;
tcuvidDestroyVideoSource              *cuvidDestroyVideoSource;
tcuvidSetVideoSourceState             *cuvidSetVideoSourceState;
tcuvidGetVideoSourceState             *cuvidGetVideoSourceState;
tcuvidGetSourceVideoFormat            *cuvidGetSourceVideoFormat;
tcuvidGetSourceAudioFormat            *cuvidGetSourceAudioFormat;

Is there any update from opencv for video decoding? How do we use FFmpeg in OpenCV to replace "dynlink_nvcuvid.h"? I am trying to use NVIDIA video decoding but I am stuck at converting the video frame to GpuMat.

I am using Ubuntu 18 Cuda 9.2 OpenCV 3.4.2

For Cuda part, to decode I have used the following code:

FFmpegDemuxer demuxer(szInFilePath);
NvDecoder dec(cuContext, demuxer.GetWidth(), demuxer.GetHeight(), true, FFmpeg2NvCodecId(demuxer.GetVideoCodec()));
FramePresenterGL presenter(cuContext, demuxer.GetWidth(), demuxer.GetHeight());
uint8_t *dpFrame = 0;
int nPitch = 0;
int nVideoBytes = 0, nFrameReturned = 0, nFrame = 0;
uint8_t *pVideo = NULL;
uint8_t **ppFrame;
do {
    demuxer.Demux(&pVideo, &nVideoBytes);
    dec.Decode(pVideo, nVideoBytes, &ppFrame, &nFrameReturned);
    if (!nFrame && nFrameReturned)
        LOG(INFO) << dec.GetVideoInfo();

    for (int i = 0; i < nFrameReturned; i++) {
        presenter.GetDeviceFrameBuffer(&dpFrame, &nPitch);
        if (dec.GetBitDepth() == 8)
            Nv12ToBgra32((uint8_t *)ppFrame[i], dec.GetWidth(), (uint8_t *)dpFrame, nPitch, dec.GetWidth(), dec.GetHeight());
        else
            P016ToBgra32((uint8_t *)ppFrame[i], 2 * dec.GetWidth(), (uint8_t *)dpFrame, nPitch, dec.GetWidth(), dec.GetHeight());

        cv::Size sz(dec.GetWidth(),dec.GetHeight());
        cv::Mat mat1(sz,CV_8UC3, ppFrame);
        cv::imshow ("test", mat1);
        cv::waitKey(0);
    }
    nFrame += nFrameReturned;
} while (nVideoBytes);

Anyone has experience on using opencv cuda with actual video codec api? How do I put the video frame into GpuMat?

Update:

I have tried to convert uint8_t but the image is displayed wrongly as shown below: enter image description here

The code i used is updated above. What should I change to display the image correctly? I tried https://stackoverflow.com/a/51520728/7939409 but the downloadedLeft is empty for the code below. 

cv::Mat downloadedLeft;
cv::cuda::GpuMat gpuLeft;

cudaMalloc((void **)&ppFrame, gpuLeft.rows*gpuLeft.step);
cudaMemcpyAsync(ppFrame, gpuLeft.ptr<uint8_t>(), gpuLeft.rows*gpuLeft.step, cudaMemcpyDeviceToDevice);

gpuLeft.download(downloadedLeft);
cv::imshow ("test", downloadedLeft);
cv::waitKey(1);
Solution

  • I found a simple c way to read an image without any 3rd party.

    #include <stdio.h>
#include <stdlib.h>
#include <assert.h>

int read_image(FILE *rfile, void *buf, int width, int height, int stride, int elem_size)
{
    char *byte_ptr = buf;
    int i;
    int ret = 1;

    if (width <= 0 || height <= 0 || elem_size <= 0)
    {
        goto fail_or_end;
    }

    for (i = 0; i < height; ++i)
    {
        if (fread(byte_ptr, elem_size, width, rfile) != (size_t)width)
        {
            goto fail_or_end;
        }

        byte_ptr += stride;
    }

    ret = 0;

fail_or_end:
    return ret;
}

    credit to netflix vmaf open source (file_io.c). Thank you netflix. After I read the image, I managed to continue process it with cuda.