How to convert a OpenCvSharp Mat to a Windows.Graphics.Imaging.SoftwareBitmap efficiently?

Question

I'm capturing images from a camera with OpenCvSharp as Mat objects. Now I'd like to show them as a preview in a WinUI3 application.

So for this, I need to convert the Mat images to a SoftwareBitmap.

I have a working solution, but as I'm basically encoding the Mat as png and decode it as SoftwareBitmap, it is pretty inefficient (pegs a single CPU core of my 5800x3D at 100% for a 30 fps 720p stream):

    private async Task<SoftwareBitmap> GetSoftwareBitmapFromMat(Mat openCvImage)
    {
        using var stream = openCvImage.ToMemoryStream();
        using var randomAccessStream = stream.AsRandomAccessStream();
        BitmapDecoder decoder = await BitmapDecoder.CreateAsync(BitmapDecoder.PngDecoderId, randomAccessStream);
        using SoftwareBitmap softwareBitmap = await decoder.GetSoftwareBitmapAsync();
        return SoftwareBitmap.Convert(softwareBitmap, BitmapPixelFormat.Bgra8, BitmapAlphaMode.Premultiplied);
    }

I've come across another potential solution in the OpenCvSharp Samples repo.

    public static unsafe void Mat2SoftwareBitmap(Mat input, SoftwareBitmap output)
    {
        using (BitmapBuffer buffer = output.LockBuffer(BitmapBufferAccessMode.ReadWrite))
        {
            using (var reference = buffer.CreateReference())
            {
                byte* dataInBytes;
                uint capacity;
                ((IMemoryBufferByteAccess)reference).GetBuffer(&dataInBytes, &capacity);
                BitmapPlaneDescription bufferLayout = buffer.GetPlaneDescription(0);

                for (int i = 0; i < bufferLayout.Height; i++)
                {
                    for (int j = 0; j < bufferLayout.Width; j++)
                    {
                        dataInBytes[bufferLayout.StartIndex + bufferLayout.Stride * i + 4 * j + 0] =
                            input.DataPointer[bufferLayout.StartIndex + bufferLayout.Stride * i + 4 * j + 0];
                        dataInBytes[bufferLayout.StartIndex + bufferLayout.Stride * i + 4 * j + 1] =
                            input.DataPointer[bufferLayout.StartIndex + bufferLayout.Stride * i + 4 * j + 1];
                        dataInBytes[bufferLayout.StartIndex + bufferLayout.Stride * i + 4 * j + 2] =
                            input.DataPointer[bufferLayout.StartIndex + bufferLayout.Stride * i + 4 * j + 2];
                        dataInBytes[bufferLayout.StartIndex + bufferLayout.Stride * i + 4 * j + 3] =
                            input.DataPointer[bufferLayout.StartIndex + bufferLayout.Stride * i + 4 * j + 3];
                    }
                }
            }
        }
    }


    [ComImport]
    [Guid("fbc4dd2d-245b-11e4-af98-689423260cf8")]
    [InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
    public unsafe interface IMemoryBufferByteAccess
    {
        void GetBuffer([Out] byte** buffer, [Out] uint* capacity);
    }

However this flat out doesn't seem to work, as I'm getting an InvalidCastException for IMemoryBufferByteAccess:

System.InvalidCastException
  HResult=0x80004002
  Message=Invalid cast from 'WinRT.IInspectable' to '<Project>+IMemoryBufferByteAccess'.

This occurs with both the original code and the code above where I've taken the IMemoryBufferByteAccess interface and its Guid from here. (And even without this exception, I'd probably have to work out which pixelformat to use for the SoftwareBitmap, as the input Mat is using CV_8UC3).

I've also tried various ways to use SoftwareBitmap.CreateCopyFromBuffer, however I always get a COMException (that is no help whatsoever):

    private SoftwareBitmap GetSoftwareBitmapFromMat(Mat openCvImage)
    {
        using Mat convertedImage = new Mat();
        Cv2.CvtColor(openCvImage, convertedImage, ColorConversionCodes.BGR2BGRA);
        var buffer = CryptographicBuffer.CreateFromByteArray(convertedImage.AsSpan<byte>().ToArray());
        return SoftwareBitmap.CreateCopyFromBuffer(buffer,
            BitmapPixelFormat.Bgra8,
            openCvImage.Width,
            openCvImage.Height, BitmapAlphaMode.Premultiplied);
    }

Answer 1

I would suggest using a WriteableBitmap instead of SoftwareBitmap. If the goal is to show images on screen the particular type of bitmap should not matter. With a WriteableBitmap you can copy pixeldata directly from a native pointer.

I'm not that familiar with how OpenCV represents memory, and I'm unsure if you should use .DataPointer, or .GetData(false), or need to rearrange data somehow. It could be as simple as:

myWriteableBitmap.WritePixels(
    new Int32Rect(0, 0, width, height),
    sourceDataPointer,
    height * stride,
    stride)

You can also get a pointer to the backbuffer of the writeable bitmap and copy pixels yourself:

self.Lock();
try
{
    var ptr = (byte*)myWriteableBitmap.BackBuffer;
    // Copy from input to ptr
    myWriteableBitmap.AddDirtyRect(new Int32Rect(0, 0, myWriteableBitmap.PixelWidth, myWriteableBitmap.PixelHeight));
}
finally
{
    myWriteableBitmap.Unlock();
}

You also want to ensure that the target bitmap uses the same pixel format as your source, otherwise you need to convert each pixel yourself.