How to let FFMPEG fetch frames from OpenCV and stream them to HTTP server

Question

There is a camera that shoots at 20 frame per second. each frame is 4000x3000 pixel. The frames are sent to a software that contain openCV in it. OpenCV resizes the freames to 1920x1080 then they must be sent to FFMPEG to be encoded to H264 or H265 using Nvidia Nvenc. The encoded video then got steamed HTTP to a maximum of 10 devices.

The infrastructure is crazy good (10 GB Lan) with state of the art switchers, routers etc...

Right now, i can get 90 FPS when encoding the images from an Nvme SSD. this means that the required encoding speed is achieved.

The question is how to get the images from OpenCV to FFMPEG ? the stream will be watched on a webapp that was made using MERN stack (assuming that this is relevant).

Answer 1

For cv::Mat you have cv::VideoWriter. If you wish to use FFMpeg, assuming Mat is continuous, which can be enforced:

if (! mat.isContinuous())
{ 
    mat = mat.clone();
}

you can simply feed mat.data into sws_scale

sws_scale(videoSampler, mat.data, stride, 0, mat.rows, videoFrame->data, videoFrame->linesize);

or directly into AVFrame

For cv::cuda::GpuMat, VideoWriter implementation is not available, but you can use NVIDIA Video Codec SDK and similarly feed cv::cuda::GpuMat::data into NvEncoderCuda, just make sure your GpuMat has 4 channels (BGRA):

NV_ENC_BUFFER_FORMAT eFormat = NV_ENC_BUFFER_FORMAT_ABGR;
std::unique_ptr<NvEncoderCuda> pEnc(new NvEncoderCuda(cuContext, nWidth, nHeight, eFormat));

...
cv::cuda::cvtColor(srcIn, srcIn, cv::ColorConversionCodes::COLOR_BG2BGRA);
NvEncoderCuda::CopyToDeviceFrame(cuContext, srcIn.data, 0, (CUdeviceptr)encoderInputFrame->inputPtr,
            (int)encoderInputFrame->pitch,
            pEnc->GetEncodeWidth(),
            pEnc->GetEncodeHeight(),
            CU_MEMORYTYPE_HOST,
            encoderInputFrame->bufferFormat,
            encoderInputFrame->chromaOffsets,
            encoderInputFrame->numChromaPlanes);

Here's my complete sample of using GpuMat with NVIDIA Video Codec SDK