Capturing video frame without saving image to use Microsoft's Face API

Question

i'm making a C# UWP project which uses the webcam and takes a picture every time it sees a face. This picture is saved on the Image folder, overwriting the previous one, and is then used on the Microsoft's Face API. What i'm trying to do now is to instead of saving the picture on the Image folder i wanted to get the frame and use it on the API call.

Already searched for a bunch of stuff but didn't find anything very specific, found some third-party class called FrameGrabber but didn't understand very well how to work with it and did not find any helpfull documentation to assist me with that.

Was wondering if anyone has anyone ideia of a way of doing this, and if so if you could provide me with any documentation or something that oculd assist me on this.

Thanks in advance.

Answer 1

Capturing video frame without saving image to use Microsoft's Face API

You could refer CameraGetPreviewFrame official code sample. It use CaptureElement to display the camera preview frame. You could get the SoftwareBitmap from _mediaCapture instance directly. Then pass this SoftwareBitmap to Microsoft's Face API.

private async Task GetPreviewFrameAsSoftwareBitmapAsync()
{
    // Get information about the preview
    var previewProperties = _mediaCapture.VideoDeviceController.GetMediaStreamProperties(MediaStreamType.VideoPreview) as VideoEncodingProperties;

    // Create the video frame to request a SoftwareBitmap preview frame
    var videoFrame = new VideoFrame(BitmapPixelFormat.Bgra8, (int)previewProperties.Width, (int)previewProperties.Height);

    // Capture the preview frame
    using (var currentFrame = await _mediaCapture.GetPreviewFrameAsync(videoFrame))
    {
        // Collect the resulting frame
        SoftwareBitmap previewFrame = currentFrame.SoftwareBitmap;

        // Show the frame information
        FrameInfoTextBlock.Text = String.Format("{0}x{1} {2}", previewFrame.PixelWidth, previewFrame.PixelHeight, previewFrame.BitmapPixelFormat);

        // Add a simple green filter effect to the SoftwareBitmap
        if (GreenEffectCheckBox.IsChecked == true)
        {
            ApplyGreenFilter(previewFrame);
        }

        // Show the frame (as is, no rotation is being applied)
        if (ShowFrameCheckBox.IsChecked == true)
        {
            // Create a SoftwareBitmapSource to display the SoftwareBitmap to the user
            var sbSource = new SoftwareBitmapSource();
            await sbSource.SetBitmapAsync(previewFrame);

            // Display it in the Image control
            PreviewFrameImage.Source = sbSource;
        }

    }
}

Update You could use the following code to converter SoftwareBitmap to IRandomAccessStream, then pass it to face detect api.

private async Task<InMemoryRandomAccessStream> EncodedStream(SoftwareBitmap soft, Guid encoderId)
{

    using (var ms = new InMemoryRandomAccessStream())
    {
        BitmapEncoder encoder = await BitmapEncoder.CreateAsync(encoderId, ms);
        encoder.SetSoftwareBitmap(soft);

        try
        {
            await encoder.FlushAsync();
        }
        catch (Exception ex)
        {

        }

        return ms;
    }
}