Creating video with MediaCapture with differing Preview and Record streams

Question

I'm trying to create a Windows Store application that allows the UI to be captured and overlaid on video coming from a web camera.

I'm using the MediaCapture class to manage the video capture process. I've created a MFT (based on the Grayscale sample) that allows me to accomplish this in a basic manner. This MFT has been added as an effect to the Record Stream of the MediaCapture class, and I'm able to create a video file with the UI overlaid on the camera video easily enough. (Easily is a relative term)

The problem that I've hit is that the overlay from the MFT is showing up in the preview stream, which is also being displayed on screen. So the UI is being displayed normally, and also in the video stream. This is a bad result, as I don't want the effect applied to the preview stream and don't want the user to see the UI in the video preview, only in the resulting recording.

Is there a way to make the MediaCapture class use the effect only on the record stream, and not the preview stream?

If there is not an easy way to do this, can this be implemented by creating a custom sink? The MediaCapture could record to the custom sink, and the custom sink would add the overlay and save to video?

Answer 1

With some cameras (USB webcams in particular), record/preview/photo all come from the same video stream. So applying an effect to one applies the effect to all. Whether video streams are identical or independent is given by MediaCapture.MediaCaptureSettings.VideoDeviceCharacteristic.

So in your case, using a custom sink seems the way to go. IMFSinkWriter can be used to encode frames after adding the overlay.

For reference, code snippet adding an effect to preview+record for any type of camera (effectively the opposite of what you are trying to do):

MediaCapture capture = ...;
await capture.AddEffectAsync(MediaStreamType.VideoPreview, "Extensions.MyEffect", null);

// If preview and record are different streams, also add the effect there
if ((capture.MediaCaptureSettings.VideoDeviceCharacteristic != VideoDeviceCharacteristic.AllStreamsIdentical) && 
    (capture.MediaCaptureSettings.VideoDeviceCharacteristic != VideoDeviceCharacteristic.PreviewRecordStreamsIdentical))
{
    await capture.AddEffectAsync(MediaStreamType.VideoRecord, "Extensions.MyEffect", null);
}