I have a ffmpeg-autogen
video decoder, im trying to use a D3D11
as HW accelerator, but the frame i get in result(with NV12
format) is broken. If im doing the same with Vulkan
as accelerator, everything works fine, but vulkan
uses more graphic card resources..
For rendering i use OpenTK, received frames i convert to RGB with textures and shaders.
Below my decoder initialization.
public VideoStreamDecoder(AVCodecParameters* parameters, AVCodec* codec)
{
AVCodec* _codec = codec;
_pCodecContext = ffmpeg.avcodec_alloc_context3(_codec);
ffmpeg.av_hwdevice_ctx_create(&_pCodecContext->hw_device_ctx, AVHWDeviceType.AV_HWDEVICE_TYPE_D3D11VA, null, null, 0).ThrowExceptionIfError();
ffmpeg.avcodec_parameters_to_context(_pCodecContext, parameters).ThrowExceptionIfError();
ffmpeg.avcodec_open2(_pCodecContext, _codec, null).ThrowExceptionIfError();
CodecName = ffmpeg.avcodec_get_name(_codec->id);
FrameSize = new Size(_pCodecContext->width, _pCodecContext->height);
PixelFormat = _pCodecContext->pix_fmt;
_pFrame = ffmpeg.av_frame_alloc();
_receivedFrame = ffmpeg.av_frame_alloc();
}
And frames reading function
public bool TryReadNextFrame(out AVFrame frame, AVPacket packet)
{
int error;
do
{
ffmpeg.avcodec_send_packet(_pCodecContext, &packet).ThrowExceptionIfError();
error = ffmpeg.avcodec_receive_frame(_pCodecContext, _pFrame);
} while (error == ffmpeg.AVERROR(ffmpeg.EAGAIN));
error.ThrowExceptionIfError();
ffmpeg.av_hwframe_transfer_data(_receivedFrame, _pFrame, 0);
//here i get NV12 frame
var clonedFrame = ffmpeg.av_frame_clone(_receivedFrame);
frame = *clonedFrame;
return true;
}
And here is result i get with both accelerators:
Vulkan
normally working
The same camera, but D3D11
broken
One more intresting moment, is that cameras, that dont work have yuv420p
format, the only one working with D3D11
has yuvj420
.
I tried changing format using sws_scale
from NV12 to NV12, and it worked, but it uses too much CPU.
Also tried changin format to rgb before av_hwframe_transfer_data
function and rendering it using Vulkan rendering, it sort of works with Vulkan
, but with D3D11
it doesnt.
UPDATE
I noticed, that frame linesize was bigger, than width and tried
_receivedFrame->linesize[0] = _receivedFrame->width;
And it helped, now image looks better, but still not perfect.
UPDATE 2
I triead also
_receivedFrame->linesize[1] = _receivedFrame->width;
and now everything works just as it should, took me whole day to write two lines of code :)
The problem was that frame linesize was actually bigger, than width. Making it equal solved the problem.
ffmpeg.av_hwframe_transfer_data(_receivedFrame, _pFrame, 0);
_receivedFrame->linesize[0] = _receivedFrame->width;
_receivedFrame->linesize[1] = _receivedFrame->width;