I'm trying to get a HEVC video decoder working with Media Foundation. What I have is working - for about half a dozen frames - and then the MFTransform::ProcessOutput function hangs indefinitely with the following callstack:
NtWaitForMultipleObjects()
WaitForMultipleObjectsEx()
CDXVAFrameManager::WaitOnSampleReturnedByRenderer(void)
CH265DecoderTransform::ProcessOutput(unsigned long,unsigned long,struct _MFT_OUTPUT_DATA_BUFFER *,unsigned long *)
I presume this is because I'm not releasing something that I should be releasing. But I have no idea what.
I have a MFTransform video decoder and video processor set up to decode HEVC data and convert from NV12 to RGB, respectively. They are connected to the DirectX 11 device used for rendering by sending the MFT_MESSAGE_SET_D3D_MANAGER message. In this configuration, they create Samples themselves (MFT_OUTPUT_STREAM_PROVIDES_SAMPLES is set in the MFT_OUTPUT_STREAM_INFO structure), and I release them when I'm done with them.
As I mention above, this is working. At least for a few frames. Video samples are decoded to NV12, converted to RGB, and displayed.
I am querying for a D3D11Multithread object and passing TRUE to SetMultithreadProtected. Which made no noticeable difference. Do I need to protect every call into DirectX with Enter/Leave calls?
I found this question here: https://social.msdn.microsoft.com/Forums/azure/en-US/fd399107-69d3-4b55-81a9-9ab6db089cfa/video-decoding-with-custom-mft-or-msft-h264-decoder-mft?forum=mediafoundationdevelopment with exactly the same issue, which was solved with the rather cryptic:
...got this working by releasing pointers that Windows had internally allocated.
But I'm at a loss as to which pointers were meant.
Prior to the decoder hanging, ProcessOutput for the decoder is only returning S_OK. If I pause the decoder right before it hangs, and just run the frame processor, it returns MF_E_SAMPLEALLOCATOR_EMPTY, which suggests something in the RGB samples isn't being cleaned up.
But what?
As far as I can tell, I have no resource leaks in my code. I am releasing all the Samples created by the Media Foundation transforms. (Attempting to forcibly release them further results in a crash when my smart pointer class attempts to release them legitimately later on.)
Do I need to query for sample buffers and release those? Or the surfaces they contain? Forcibly removing the sample buffers by calling RemoveAllBuffers made no difference.
I have noticed that the decoder returns a CPooledSample sample, while the video processor returns a CMFTrackedSample, so is the decoder not being notified a sample has been released? Is that something I should be doing somehow?
Edit: A copy of the relevant parts of my code can be found here: https://pastebin.com/Y6p30MRG
OK, it was (as I'm sure is no surprise to anyone) a stupid mistake on my part, and a monumentally poor reaction from Media Foundation to my mistake:
I wasn't calling MFStartup.
But the fact Media Foundation didn't bother to tell me this, and just pretended everything was fine (until it wasn't) is, frankly, an embarrassingly bad "design".
Once I started calling MFStartup, everything else began working as expected.