I'm trying to encode the ID3D11Texture2D to mp4 using MediaFoundation. Below is my current code.
Initializing Sink Writer
private int InitializeSinkWriter(String outputFile, int videoWidth, int videoHeight)
{
IMFMediaType mediaTypeIn = null;
IMFMediaType mediaTypeOut = null;
IMFAttributes attributes = null;
int hr = 0;
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateAttributes(out attributes, 1);
if (Succeeded(hr)) hr = (int)attributes.SetUINT32(MFAttributesClsid.MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, 1);
if (Succeeded(hr)) hr = (int)attributes.SetUINT32(MFAttributesClsid.MF_LOW_LATENCY, 1);
// Create the sink writer
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateSinkWriterFromURL(outputFile, null, attributes, out sinkWriter);
// Create the output type
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateMediaType(out mediaTypeOut);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetGUID(MFAttributesClsid.MF_MT_MAJOR_TYPE, MFMediaType.Video);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetGUID(MFAttributesClsid.MF_TRANSCODE_CONTAINERTYPE, MFTranscodeContainerType.MPEG4);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetGUID(MFAttributesClsid.MF_MT_SUBTYPE, MFMediaType.H264);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetUINT32(MFAttributesClsid.MF_MT_AVG_BITRATE, videoBitRate);
if (Succeeded(hr)) hr = (int)mediaTypeOut.SetUINT32(MFAttributesClsid.MF_MT_INTERLACE_MODE, (int)MFVideoInterlaceMode.Progressive);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeSize(mediaTypeOut, MFAttributesClsid.MF_MT_FRAME_SIZE, videoWidth, videoHeight);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeRatio(mediaTypeOut, MFAttributesClsid.MF_MT_FRAME_RATE, VIDEO_FPS, 1);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeRatio(mediaTypeOut, MFAttributesClsid.MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
if (Succeeded(hr)) hr = (int)sinkWriter.AddStream(mediaTypeOut, out streamIndex);
// Create the input type
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateMediaType(out mediaTypeIn);
if (Succeeded(hr)) hr = (int)mediaTypeIn.SetGUID(MFAttributesClsid.MF_MT_MAJOR_TYPE, MFMediaType.Video);
if (Succeeded(hr)) hr = (int)mediaTypeIn.SetGUID(MFAttributesClsid.MF_MT_SUBTYPE, MFMediaType.ARGB32);
if (Succeeded(hr)) hr = (int)mediaTypeIn.SetUINT32(MFAttributesClsid.MF_SA_D3D11_AWARE, 1);
if (Succeeded(hr)) hr = (int)mediaTypeIn.SetUINT32(MFAttributesClsid.MF_MT_INTERLACE_MODE, (int)MFVideoInterlaceMode.Progressive);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeSize(mediaTypeIn, MFAttributesClsid.MF_MT_FRAME_SIZE, videoWidth, videoHeight);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeRatio(mediaTypeIn, MFAttributesClsid.MF_MT_FRAME_RATE, VIDEO_FPS, 1);
if (Succeeded(hr)) hr = (int)MFExtern.MFSetAttributeRatio(mediaTypeIn, MFAttributesClsid.MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
if (Succeeded(hr)) hr = (int)sinkWriter.SetInputMediaType(streamIndex, mediaTypeIn, null);
// Start accepting data
if (Succeeded(hr)) hr = (int)sinkWriter.BeginWriting();
COMBase.SafeRelease(mediaTypeOut);
COMBase.SafeRelease(mediaTypeIn);
return hr;
}
Writing frame
int hr = 0;
IMFSample sample = null;
IMFMediaBuffer buffer = null;
IMF2DBuffer p2Dbuffer = null;
object texNativeObject = Marshal.GetObjectForIUnknown(surface.NativePointer);
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateDXGISurfaceBuffer(new Guid("6f15aaf2-d208-4e89-9ab4-489535d34f9c"), texNativeObject, 0, false, out p2Dbuffer);
buffer = MFVideoEncoderST.ReinterpretCast<IMF2DBuffer,IMFMediaBuffer>(p2Dbuffer);
int length=0;
if (Succeeded(hr)) hr = (int)p2Dbuffer.GetContiguousLength(out length);
if (Succeeded(hr)) hr = (int)buffer.SetCurrentLength(length);
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateVideoSampleFromSurface(null, out sample);
if (Succeeded(hr)) hr = (int)sample.AddBuffer(buffer);
if (Succeeded(hr)) hr = (int)sample.SetSampleTime(prevRecordingDuration);
if (Succeeded(hr)) hr = (int)sample.SetSampleDuration((recordDuration - prevRecordingDuration));
if (Succeeded(hr)) hr = (int)sinkWriter.WriteSample(streamIndex, sample);
COMBase.SafeRelease(sample);
COMBase.SafeRelease(buffer);
using MFTRACE I'm getting the below error.
02:48:04.99463 CMFSinkWriterDetours::WriteSample @024BEA18 Stream Index 0x0, Sample @17CEACE0, Time 571ms, Duration 16ms, Buffers 1, Size 4196352B,2088,2008 02:48:04.99465 CMFSinkWriterDetours::WriteSample @024BEA18 failed hr=0x887A0005 (null)2088,2008
02:48:05.01090 CMFSinkWriterDetours::WriteSample @024BEA18 Stream Index 0x0, Sample @17CE9FC0, Time 587ms, Duration 17ms, Buffers 1, Size 4196352B,2088,2008 02:48:05.01091 CMFSinkWriterDetours::WriteSample @024BEA18 failed hr=0x887A0005 (null)2088,2008
02:48:05.02712 CMFSinkWriterDetours::WriteSample @024BEA18 Stream Index 0x0, Sample @17CEACE0, Time 604ms, Duration 16ms, Buffers 1, Size 4196352B,2088,2008 02:48:05.02713 CMFSinkWriterDetours::WriteSample @024BEA18 failed hr=0x887A0005 (null)
Can anyone tell me whats wrong with my code? I can only produce 0 bytes of mp4 file.
There are a few potential problems that occur to me here. Roman mentioned the two big ones so I'll elaborate on those. I have a couple other critiques / suggestions for you as well.
IMFDXGIDeviceManagerIn order to use hardware acceleration within Media Foundation, you need to create a DirectX device manager object, either an IDirect3DDeviceManager9 for DX9 or in your case an IMFDXGIDeviceManager for DXGI. I strongly suggest reading all the MSDN documentation of that interface. The reason this is necessary is because the same DX device must be shared across all the cooperating hardware MF transforms being used, since they all need access to the shared GPU memory the device controls, and each one needs exclusive control of the device while it's working, so a locking system is needed. The device manager object provides that locking system, and is also the standard way of providing a DX device to one or more transforms. For DXGI, you create this using MFCreateDXGIDeviceManager.
From there, you need to create your DX11 device, and call IMFDXGIDeviceManager::ResetDevice with your DX11 device. You then need to set the device manager for the Sink Writer itself, which is not done in the code you provided above. That is accomplished like this:
// ... inside your InitializeSinkWriter function that you listed above
// I'm assuming you've already created and set up the DXGI device manager elsewhere
IMFDXGIDeviceManager pDeviceManager;
// Passing 3 as the argument because we're adding 3 attributes immediately, saves re-allocations
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateAttributes(out attributes, 3);
if (Succeeded(hr)) hr = (int)attributes.SetUINT32(MFAttributesClsid.MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, 1);
if (Succeeded(hr)) hr = (int)attributes.SetUINT32(MFAttributesClsid.MF_LOW_LATENCY, 1);
// Here's the key piece!
if (Succeeded(hr)) hr = (int)attributes.SetUnknown(MFAttributesClsid.MF_SINK_WRITER_D3D_MANAGER, pDeviceManager);
// Create the sink writer
if (Succeeded(hr)) hr = (int)MFExtern.MFCreateSinkWriterFromURL(outputFile, null, attributes, out sinkWriter);
This will actually enable D3D11 support for the hardware encoder and allow it access to read the Texture2D you're passing in. It's worth noting that MF_SINK_WRITER_D3D_MANAGER works for both DX9 and DXGI device managers.
IMFSample instances of the same textureThis is a potential cause of your problem as well - at the very least it will lead to a lot of unintended behavior even if it isn't the cause of the obvious problem. Building off Roman's comment, many encoders will buffer multiple frames as part of their encoding process. You don't see that behavior when using the Sink Writer because it handles all the detail work for you. However, what you are trying to accomplish (i.e. sending D3D11 textures as input frames) is sufficiently low level that you start having to worry about the internal details of the encoder MFT being used by the Sink Writer.
Most video encoder MFTs will use an internal buffer of some size to store the last N samples provided via IMFTransform::ProcessInput. This has the side effect that multiple samples must be provided as inputs before any output will be generated. Video encoders need access to multiple samples in order because they use the subsequent frames to determine how to encode the current frame. In other words, if the decoder is working on frame 0, it might need to see frames 1, 2, and 3 as well. From a technical standpoint, this is because of things like inter-frame prediction and motion estimation. Once the encoder is finished processing the oldest sample, it generates an output buffer (another IMFSample object, but this time on the output side via IMFTransform::ProcessOutput) then discards the input sample it was working on (by calling IUnknown::Release), then requests more input, and eventually moves on to the next frame. You can read more about this process in the MSDN article Processing Data in the Encoder
What this means, as Roman alluded to, is that you are encapsulating an ID3D11Texture2D inside an IMFMediaBuffer inside an IMFSample and then passing that to the Sink Writer. That sample is likely being buffered by the encoder as part of the encoding process. As the encoder is working, the contents of that Texture2D are probably changing, which can cause a variety of problems. Even if that didn't cause program errors, it would certainly lead to very strange encoded video outputs. Imagine if the encoder is trying to predict how the visual content of one frame changes in the next frame, and then the actual visual content of both frames is updated out from under the encoder!
This specific problem is happening because the encoder only has a pointer reference to your IMFSample instance, which is ultimately just a pointer itself to your ID3D11Texture2D object, and that object is a kind of pointer reference to mutable graphics memory. Ultimately, the contents of that graphics memory are changing due to some other part of your program, but because it's always the same GPU texture being updated, every single sample you send the encoder points to the same single texture. That means whenever you update the texture, by changing GPU memory, all the active IMFSample objects will reflect those changes, since they are all effectively pointing to the same GPU texture.
To fix this, you'll need to allocate multiple ID3D11Texture2D objects, such that you can pair up one texture with one IMFSample when providing it to the Sink Writer. This will fix the problem of all the samples pointing to the same single GPU texture by making each sample point to a unique texture. You won't necessarily know how many textures you need to create, though, so the safest way to handle this is to write your own texture allocator. This can still be done within C# for what it's worth, MediaFoundation.NET has the interfaces defined that you'll need to use.
The allocator should maintain a list of "free" SharpDX.Texture2D objects - those that are not currently in use by the Sink Writer / encoder. Your program should be able to request new texture objects from the allocator, in which case it will either return an object from the free list, or create a new texture to accommodate the request.
The next problem is knowing when the IMFSample object has been discarded by the encoder, so that you can add the attached texture back to the free list. As it happens, the MFCreateVideoSampleFromSurface function you're currently using allocates samples that implement the IMFTrackedSample interface. You'll need that interface so that you can be notified when the sample is freed, so that you can reclaim the Texture2D objects.
The trick is that you have to tell the sample that you are the allocator. First, your allocator class needs to implement IMFAsyncCallback. If you set your allocator class on the sample via IMFTrackedSample::SetAllocator, your allocator's IMFAsyncCallback::Invoke method will be called, with an IMFAsyncResult passed as an argument whenever the encoder releases the sample. Here's a general example of what that allocator class could look like.
sealed class TextureAllocator : IMFAsyncCallback, IDisposable
{
private ConcurrentStack<SharpDX.Direct3D11.Texture2D> m_freeStack;
private static readonly Guid s_IID_ID3D11Texture2D = new Guid("6f15aaf2-d208-4e89-9ab4-489535d34f9c");
// If all textures are the exact same size and color format,
// consider making those parameters private class members and
// requiring they be specified as arguments to the constructor.
public TextureAllocator()
{
m_freeStack = new ConcurrentStack<SharpDX.Direct3D11.Texture2D>();
}
private bool disposedValue = false;
private void Dispose(bool disposing)
{
if(!disposedValue)
{
if(disposing)
{
// Dispose managed resources here
}
if(m_freeStack != null)
{
SharpDX.Direct3D11.Texture2D texture;
while(m_freeStack.TryPop(out texture))
{
texture.Dispose();
}
m_freeStack = null;
}
disposedValue = true;
}
}
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
~TextureAllocator()
{
Dispose(false);
}
private SharpDX.Direct3D11.Texture2D InternalAllocateNewTexture()
{
// Allocate a new texture with your format, size, etc here.
}
public SharpDX.Direct3D11.Texture2D AllocateTexture()
{
SharpDX.Direct3D11.Texture2D existingTexture;
if(m_freeStack.TryPop(out existingTexture))
{
return existingTexture;
}
else
{
return InternalAllocateNewTexture();
}
}
public IMFSample CreateSampleAndAllocateTexture()
{
IMFSample pSample;
IMFTrackedSample pTrackedSample;
HResult hr;
// Create the video sample. This function returns an IMFTrackedSample per MSDN
hr = MFExtern.MFCreateVideoSampleFromSurface(null, out pSample);
MFError.ThrowExceptionForHR(hr);
// Query the IMFSample to see if it implements IMFTrackedSample
pTrackedSample = pSample as IMFTrackedSample;
if(pTrackedSample == null)
{
// Throw an exception if we didn't get an IMFTrackedSample
// but this shouldn't happen in practice.
throw new InvalidCastException("MFCreateVideoSampleFromSurface returned a sample that did not implement IMFTrackedSample");
}
// Use our own class to allocate a texture
SharpDX.Direct3D11.Texture2D availableTexture = AllocateTexture();
// Convert the texture's native ID3D11Texture2D pointer into
// an IUnknown (represented as as System.Object)
object texNativeObject = Marshal.GetObjectForIUnknown(availableTexture.NativePointer);
// Create the media buffer from the texture
IMFMediaBuffer p2DBuffer;
hr = MFExtern.MFCreateDXGISurfaceBuffer(s_IID_ID3D11Texture2D, texNativeObject, 0, false, out p2DBuffer);
// Release the object-as-IUnknown we created above
COMBase.SafeRelease(texNativeObject);
// If media buffer creation failed, throw an exception
MFError.ThrowExceptionForHR(hr);
// Set the owning instance of this class as the allocator
// for IMFTrackedSample to notify when the sample is released
pTrackedSample.SetAllocator(this, null);
// Attach the created buffer to the sample
pTrackedSample.AddBuffer(p2DBuffer);
return pTrackedSample;
}
// This is public so any textures you allocate but don't make IMFSamples
// out of can be returned to the allocator manually.
public void ReturnFreeTexture(SharpDX.Direct3D11.Texture2D freeTexture)
{
m_freeStack.Push(freeTexture);
}
// IMFAsyncCallback.GetParameters
// This is allowed to return E_NOTIMPL as a way of specifying
// there are no special parameters.
public HResult GetParameters(out MFAsync pdwFlags, out MFAsyncCallbackQueue pdwQueue)
{
pdwFlags = MFAsync.None;
pdwQueue = MFAsyncCallbackQueue.Standard;
return HResult.E_NOTIMPL;
}
public HResult Invoke(IMFAsyncResult pResult)
{
object pUnkObject;
IMFSample pSample = null;
IMFMediaBuffer pBuffer = null;
IMFDXGIBuffer pDXGIBuffer = null;
// Get the IUnknown out of the IMFAsyncResult if there is one
HResult hr = pResult.GetObject(out pUnkObject);
if(Succeeded(hr))
{
pSample = pUnkObject as IMFSample;
}
if(pSample != null)
{
// Based on your implementation, there should only be one
// buffer attached to one sample, so we can always grab the
// first buffer. You could add some error checking here to make
// sure the sample has a buffer count that is 1.
hr = pSample.GetBufferByIndex(0, out pBuffer);
}
if(Succeeded(hr))
{
// Query the IMFMediaBuffer to see if it implements IMFDXGIBuffer
pDXGIBuffer = pBuffer as IMFDXGIBuffer;
}
if(pDXGIBuffer != null)
{
// Got an IMFDXGIBuffer, so we can extract the internal
// ID3D11Texture2D and make a new SharpDX.Texture2D wrapper.
hr = pDXGIBuffer.GetResource(s_IID_ID3D11Texture2D, out pUnkObject);
}
if(Succeeded(hr))
{
// If we got here, pUnkObject is the native D3D11 Texture2D as
// a System.Object, but it's unlikely you have an interface
// definition for ID3D11Texture2D handy, so we can't just cast
// the object to the proper interface.
// Happily, SharpDX supports wrapping System.Object within
// SharpDX.ComObject which makes things pretty easy.
SharpDX.ComObject comWrapper = new SharpDX.ComObject(pUnkObject);
// If this doesn't work, or you're using something like SlimDX
// which doesn't support object wrapping the same way, the below
// code is an alternative way.
/*
IntPtr pD3DTexture2D = Marshal.GetIUnknownForObject(pUnkObject);
// Create your wrapper object here, like this for SharpDX
SharpDX.ComObject comWrapper = new SharpDX.ComObject(pD3DTexture2D);
// or like this for SlimDX
SlimDX.Direct3D11.Texture2D.FromPointer(pD3DTexture2D);
Marshal.Release(pD3DTexture2D);
*/
// You might need to query comWrapper for a SharpDX.DXGI.Resource
// first, then query that for the SharpDX.Direct3D11.Texture2D.
SharpDX.Direct3D11.Texture2D texture = comWrapper.QueryInterface<SharpDX.Direct3D11.Texture2D>();
if(texture != null)
{
// Now you can add "texture" back to the allocator's free list
ReturnFreeTexture(texture);
}
}
}
}
MF_SA_D3D_AWARE on the Sink Writer input media typeI don't think this is causing the bad HRESULT you're getting, but it isn't the right thing to do regardless. MF_SA_D3D_AWARE (and its DX11 counterpart, MF_SA_D3D11_AWARE) are attributes set by an IMFTransform object to inform you that the transform supports graphics acceleration via DX9 or DX11, respectively. There is no need to set this on the Sink Writer's input media type.
SafeRelease on texNativeObjectI'd recommend calling COMBase.SafeRelease() on texNativeObject or you may leak memory. That, or you'll prolong the lifetime of that COM object unnecessarily until the GC cleans up the reference count for you
This is part of your code from above:
buffer = MFVideoEncoderST.ReinterpretCast<IMF2DBuffer,IMFMediaBuffer>(p2Dbuffer);
int length=0;
if (Succeeded(hr)) hr = (int)p2Dbuffer.GetContiguousLength(out length);
if (Succeeded(hr)) hr = (int)buffer.SetCurrentLength(length);
I'm not sure what your ReinterpretCast function is doing, but if you do need to perform a QueryInterface style cast in C#, you can just use the as operator or a regular cast.
// pMediaBuffer is of type IMFMediaBuffer and has been created elsewhere
IMF2DBuffer p2DBuffer = pMediaBuffer as IMF2DBuffer;
if(p2DBuffer != null)
{
// pMediaBuffer is an IMFMediaBuffer that also implements IMF2DBuffer
}
else
{
// pMediaBuffer does not implement IMF2DBuffer
}