Rendering video in Direct3D obtained from Media Foundation efficiently

Question

I want to use live video I am decoding from media foundation efficiently. Originally, I was running the render functions synchronously after decoding each frame. The incoming framerate is of around 25-30 fps, but I would like to render the graphics (game) content at 60fps. If I do it asynchronously I will either get corrupted output / black screens / both or very low framerate due to aggressive locking. Since the GPU operations are async I haven't been able to find a reasonable critical section. How is this normally done? I can use one of my temporary surfaces (source, dest, or g_pDecodedTexture) as a synchronization point and surround writes to it/them with a CRITICAL_SECTION, but I don't know where the critical section should go on the render (reading) thread. If I surround the whole render function, my framerate is very low, and if I don't I get incorrect output. Maybe there is another more appropriated method for synchronization.

At render setup time

    hr = g_d3dDevice->CreateShaderResourceView(g_pDecodedTexture, &shaderResourceViewDesc, &g_pTextureRV);

In the decode thread

      void Decode()
      {
           MFT_OUTPUT_DATA_BUFFER output = { 0 };
           //...
           encoder->ProcessOutput(0,1,&output,&status);
           // 
           CComPtr<IMFMediaBuffer>  spMediaBuffer;
           CComPtr<IMFDXGIBuffer>   spDXGIBuffer;
           CComPtr<IDXGIResource>  spDecodedTexture;

           output.pSample->GetBufferByIndex(0, &spMediaBuffer);
           spMediaBuffer->QueryInterface(IID_PPV_ARGS(&spDXGIBuffer);
           spDXGIBuffer->GetResource(IID_PPV_ARGS(&spDecodedTexture);

           //....

            CComPtr<ID3D11Texture2D> source;
            spDXGIBuffer->QueryInterface<ID3D11Texture2D>(&source);

            //
            CComPtr<ID3D11Resource> dest;
            swapChain->GetBuffer(0, __uuidof(ID3D11Resource), (void**)&dest);
            deviceContext->CopyResource(dest, source);
            deviceContext->CopyResource(g_pDecodedTexture, source);          
       }

In the render thread

void Render()
{
    //...
    deviceContext->PSSetShaderResources(0, 1, &g_pTextureRV);
    //..

    m_deviceContext->VSSetShaderResources(0, 1, &g_pTextureRV);
    //..
    immediateContext->DrawIndexed(...);   
    //..
    immediateContext->DrawIndexed(...);   
    //..
    immediateContext->DrawIndexed(...);   
    //..
    immediateContext->DrawIndexed(...);   
    //
    Present();
}

Answer 1

You can try this : insert the Frame Rate Converter DSP after the decoder. Be sure your input format is compatible with the DSP. Set frame rate at 60 fps.

Doing this, i think you can keep the synchronous approach.

If you want to manually display at 60 fps, we need more code to see where the problem comes from.