Search code examples
h.264video-processingms-media-foundation

Reading YUY2 data from an IMFSample that appears to have improper data on Windows 10

I am developing an application that using IMFSourceReader to read data from video files. I am using DXVA for improved performance. I am having trouble with one specific full-HD H.264 encoded AVI file. Based on my investigation this far, I believe that the IMFSample contains incorrect data. My workflow is below:

  1. Create a source reader with a D3D manager to enable hardware acceleration.
  2. Set the current media type to YUY2 as DXVA does not decode to any RGB colorspace.
  3. Call ReadSample to get an IMFSample. Works fine.
  4. Use the VideoProcessorBlt to perform YUY2 to BGRA32 conversion. For this specific file it errors out with an E_INVALIDARGS error code. Decided to do the conversion myself.
  5. Used IMFSample::ConvertToContiguousBuffer to receive an IMFMediaBuffer. When locking this buffer, the pitch is reported as 1280 bytes. This I believe is incorrect, because for a full HD video, the pitch should be (1920 + 960 + 960 = 3840 bytes).

I dumped the raw memory and extracted the Y, U and V components based on my understanding of the YUY2 layout. You can find it below. So, the data is there but I do not believe it is laid out as YUY2. Need some help in interpreting the data.

Y component

U component

V component My code for reading is below:

    // Direct3D surface that stores the result of the YUV2RGB conversion
CComPtr<IDirect3DSurface9> _pTargetSurface;


IDirectXVideoAccelerationService* vidAccelService;
initVidAccelerator(&vidAccelService); // Omitting the code for this.

// Create a new surface for doing the color conversion, set it up to store X8R8G8B8 data.
hr = vidAccelService->CreateSurface( static_cast<UINT>( 1920 ),
                                     static_cast<UINT>( 1080 ),
                                     0,                                    // no back buffers
                                     D3DFMT_X8R8G8B8,                 // data format
                                     D3DPOOL_DEFAULT,                      // default memory pool
                                     0,                                    // reserved
                                     DXVA2_VideoProcessorRenderTarget,     // to use with the Blit operation
                                     &_pTargetSurface,                     // surface used to store frame
                                     NULL);


GUID processorGUID;
DXVA2_VideoDesc videoDescriptor;
D3DFORMAT processorFmt;
UINT numSubStreams;

IDirectXVideoProcessor* _vpd;
initVideoProcessor(&vpd); // Omitting the code for this

// We get the videoProcessor parameters on creation, and fill up the videoProcessBltParams accordingly.
_vpd->GetCreationParameters(&processorGUID, &videoDescriptor, &processorFmt, &numSubStreams);

RECT targetRECT; // { 0, 0, width, height } as left, top, right, bottom
targetRECT.left = 0;
targetRECT.right = videoDescriptor.SampleWidth;
targetRECT.top = 0;
targetRECT.bottom = videoDescriptor.SampleHeight;
SIZE targetSIZE; // { width, height }
targetSIZE.cx = videoDescriptor.SampleWidth;
targetSIZE.cy = videoDescriptor.SampleHeight;

// Parameters that are required to use the video processor to perform 
// YUV2RGB and other video processing operations
DXVA2_VideoProcessBltParams _frameBltParams;

_frameBltParams.TargetRect = targetRECT;
_frameBltParams.ConstrictionSize = targetSIZE;
_frameBltParams.StreamingFlags = 0; // reserved.

_frameBltParams.BackgroundColor.Y = 0x0000;
_frameBltParams.BackgroundColor.Cb = 0x0000;
_frameBltParams.BackgroundColor.Cr = 0x0000;
_frameBltParams.BackgroundColor.Alpha = 0xFFFF;

// copy attributes from videoDescriptor obtained above.
_frameBltParams.DestFormat.VideoChromaSubsampling = videoDescriptor.SampleFormat.VideoChromaSubsampling;
_frameBltParams.DestFormat.NominalRange = videoDescriptor.SampleFormat.NominalRange;
_frameBltParams.DestFormat.VideoTransferMatrix = videoDescriptor.SampleFormat.VideoTransferMatrix;
_frameBltParams.DestFormat.VideoLighting = videoDescriptor.SampleFormat.VideoLighting;
_frameBltParams.DestFormat.VideoPrimaries = videoDescriptor.SampleFormat.VideoPrimaries;
_frameBltParams.DestFormat.VideoTransferFunction = videoDescriptor.SampleFormat.VideoTransferFunction;

_frameBltParams.DestFormat.SampleFormat = DXVA2_SampleProgressiveFrame;

// The default values are used for all these parameters.
DXVA2_ValueRange pRangePABrightness;
_vpd->GetProcAmpRange(DXVA2_ProcAmp_Brightness, &pRangePABrightness);
DXVA2_ValueRange pRangePAContrast;
_vpd->GetProcAmpRange(DXVA2_ProcAmp_Contrast, &pRangePAContrast);
DXVA2_ValueRange pRangePAHue;
_vpd->GetProcAmpRange(DXVA2_ProcAmp_Hue, &pRangePAHue);
DXVA2_ValueRange pRangePASaturation;
_vpd->GetProcAmpRange(DXVA2_ProcAmp_Saturation, &pRangePASaturation);
_frameBltParams.ProcAmpValues = { pRangePABrightness.DefaultValue, pRangePAContrast.DefaultValue,
    pRangePAHue.DefaultValue, pRangePASaturation.DefaultValue };

_frameBltParams.Alpha = DXVA2_Fixed32OpaqueAlpha();
_frameBltParams.DestData = DXVA2_SampleData_TFF;

// Input video sample for the Blt operation
DXVA2_VideoSample _frameVideoSample;

_frameVideoSample.SampleFormat.VideoChromaSubsampling = videoDescriptor.SampleFormat.VideoChromaSubsampling;
_frameVideoSample.SampleFormat.NominalRange = videoDescriptor.SampleFormat.NominalRange;
_frameVideoSample.SampleFormat.VideoTransferMatrix = videoDescriptor.SampleFormat.VideoTransferMatrix;
_frameVideoSample.SampleFormat.VideoLighting = videoDescriptor.SampleFormat.VideoLighting;
_frameVideoSample.SampleFormat.VideoPrimaries = videoDescriptor.SampleFormat.VideoPrimaries;
_frameVideoSample.SampleFormat.VideoTransferFunction = videoDescriptor.SampleFormat.VideoTransferFunction;

_frameVideoSample.SrcRect = targetRECT;
_frameVideoSample.DstRect = targetRECT;
_frameVideoSample.PlanarAlpha = DXVA2_Fixed32OpaqueAlpha();
_frameVideoSample.SampleData = DXVA2_SampleData_TFF;


CComPtr<IMFSample> sample; // Assume that this was read in from a call to ReadSample

CComPtr<IMFMediaBuffer> buffer;
HRESULT hr = sample->GetBufferByIndex(0, &buffer);
CComPtr<IDirect3DSurface9>  pSrcSurface;

// From the MediaBuffer, we get the Source Surface using MFGetService
hr = MFGetService( buffer, MR_BUFFER_SERVICE, __uuidof(IDirect3DSurface9), (void**)&pSrcSurface );

// Update the videoProcessBltParams with frame specific values.
LONGLONG sampleStartTime;
sample->GetSampleTime(&sampleStartTime);
_frameBltParams.TargetFrame = sampleStartTime;

LONGLONG sampleDuration;
sample->GetSampleDuration(&sampleDuration);

_frameVideoSample.Start = sampleStartTime;
_frameVideoSample.End = sampleStartTime + sampleDuration;
_frameVideoSample.SrcSurface = pSrcSurface;

// Run videoProcessBlt using the parameters setup (this is used for color conversion)
// The returned code is E_INVALIDARGS
hr = _vpd->VideoProcessBlt( _pTargetSurface,    // target surface
                            &_frameBltParams,   // parameters
                            &_frameVideoSample, // video sample structure
                            1,                      // one sample
                            NULL);                  // reserved
Solution

  • After a call to ReadSample of IMFSourceReader or inside OnReadSample callback function of the IMFSourceReaderCallback implementation, you might receive the MF_SOURCE_READERF_CURRENTMEDIATYPECHANGED flag. It means that the current media type has changed for one or more streams. To get the current media type call the IMFSourceReader::GetCurrentMediaType method. In your case you need to query (again) the IMFMediaType's MF_MT_FRAME_SIZE attribute for the video stream, in order to obtain the new correct video resolution. You should use the new video resolution to set the "width" and "height" values of the VideoProcessorBlt parameters' source and destination rectangles.