How to add contents of CanvasControl to a MediaClip

Question

I'm creating a uwp app where text a user types in is converted into a video overlay. The user loads the video, types in their text, gets a preview of the video with the text, then they can save their video with the overlay. Right now it works with an image overlay, but not text. I'm trying to use win2d to format the text. However, I can't figure out how to get the win2d canvas control into a windows.media.editing media clip.

I thought I could go CanvasControl > Bitmap of some sort > Image > MediaClip, but media clips can only be created through storage files or direct 3d surfaces. Anything would be helpful! This is my first time trying to write something outside of a game engine.

The part that currently converts the canvas control to a bitmap:

private async void btnAddOverlay_Click(object sender, RoutedEventArgs e)
        {

            var bitmap = new RenderTargetBitmap();
            await bitmap.RenderAsync(canvasControl);
            IBuffer pixelBuffer = await bitmap.GetPixelsAsync();
            byte[] pixels = pixelBuffer.ToArray();
            SoftwareBitmap outputBitmap = SoftwareBitmap.CreateCopyFromBuffer
            (
                pixelBuffer,
                BitmapPixelFormat.Bgra8,
                bitmap.PixelWidth,
                bitmap.PixelHeight
            );
        }

The old part that added the image overlay to the video. (overlayImageFile is the storage file. I just don't know how to convert outputBitmap to a storage file to make a media clip from it.)

private async void CreateOverlays()
        {
            // Create a clip from video and add it to the composition
            var baseVideoClip = await MediaClip.CreateFromFileAsync(pickedFile);
            composition = new MediaComposition();
            composition.Clips.Add(baseVideoClip);

            // Create a clip from image
            var overlayImageClip = await MediaClip.CreateFromImageFileAsync(overlayImageFile,timeSpan);

            // Put image in upper left corner, retain its native aspect ratio
            Rect imageOverlayPosition;
            imageOverlayPosition.Height = mediaElement.ActualHeight / 3;
            imageOverlayPosition.Width = (double)imageBitmap.PixelWidth / (double)imageBitmap.PixelHeight * imageOverlayPosition.Height;
            imageOverlayPosition.X = 0;
            imageOverlayPosition.Y = 0;

            // Make the clip from the image an overlay
            var imageOverlay = new MediaOverlay(overlayImageClip);
            imageOverlay.Position = imageOverlayPosition;
            imageOverlay.Opacity = 0.8;

            // Make a new overlay layer and add the overlay to it
            var overlayLayer = new MediaOverlayLayer();
            overlayLayer.Overlays.Add(imageOverlay);
            composition.OverlayLayers.Add(overlayLayer);

            //Render to MediaElement
            mediaElement.Position = TimeSpan.Zero;
            mediaStreamSource = composition.GeneratePreviewMediaStreamSource((int)mediaElement.ActualWidth, (int)mediaElement.ActualHeight);
            mediaElement.SetMediaStreamSource(mediaStreamSource);
            txbNotification.Text = "Overlay creaeted";
        }

Windows MediaClip Class: https://learn.microsoft.com/en-us/uwp/api/windows.media.editing.mediaclip?view=winrt-19041

Answer 1

I just don't know how to convert outputBitmap to a storage file to make a media clip from it.

After getting the RenderTargetBitmap, there is no need to convert to SoftwareBitmap. You can store the pixel data as a file in a specified format through BitmapEncoder. This is the method:

private async void btnAddOverlay_Click(object sender, RoutedEventArgs e)
{
    RenderTargetBitmap renderTargetBitmap = new RenderTargetBitmap();
    await renderTargetBitmap.RenderAsync(canvasControl);

    var pixelBuffer = await renderTargetBitmap.GetPixelsAsync();
    var pixels = pixelBuffer.ToArray();
    var displayInformation = DisplayInformation.GetForCurrentView();

    // Create the file in local folder
    var file = await ApplicationData.Current.LocalFolder.CreateFileAsync("TempCanvas.png", CreationCollisionOption.ReplaceExisting);
    
    // Write data
    using (var stream = await file.OpenAsync(FileAccessMode.ReadWrite))
    {
        var encoder = await BitmapEncoder.CreateAsync(BitmapEncoder.PngEncoderId, stream);
        encoder.SetPixelData(BitmapPixelFormat.Bgra8, BitmapAlphaMode.Premultiplied, (uint)renderTargetBitmap.PixelWidth, (uint)renderTargetBitmap.PixelHeight, displayInformation.RawDpiX, displayInformation.RawDpiY, pixels);
        await encoder.FlushAsync();
    }
}

When needed, you can use the following code to get the saved picture:

var file = await ApplicationData.Current.LocalFolder.GetFileAsync("TempCanvas.png");