I have a computer vision plugin using Hololens locatable camera on Unity 5.2. I use the class UnityEngine.XR.WSA.WebCam.PhotoCaptureFrame
with photo mode enabled at highest resolution, which is 2048x1156, in NV12 color format which is camera native format.
PhotoCaptureFrame class : https://docs.unity3d.com/2017.2/Documentation/ScriptReference/XR.WSA.WebCam.PhotoCaptureFrame.html
At the moment, I do this for my plugin to process any single photo:
PhotoCaptureFrame photo; // created correctly through Unity API
List<byte> photoBuffer = new List<byte>();
photo.CopyRawImageDataIntoBuffer(photoBuffer);
// marshal photoBuffer to plugin (and it works fine for the computer vision part)
However, it copies the raw buffer each time. I want to use the raw buffer directly from my plugin. So I tried this:
IntPtr rawBufferPtr = photo.GetUnsafePointerToBuffer(); // COM pointer to IMFMediaBuffer is incremented
// send rawBufferPtr to plugin (native function described below)
Marshal.Release(rawBufferPtr);
IMFMediaBuffer interface : https://msdn.microsoft.com/en-us/library/windows/desktop/ms696261(v=vs.85).aspx
And in my C++ plugin:
#include <Mfidl.h> // for IMFMediaBuffer support
void process_photo(void *photo_buffer_wrapper, int photo_width, int photo_height)
{
// photo_buffer_wrapper = rawBufferPtr in managed environment
IMFMediaBuffer *media_buffer = reinterpret_cast<IMFMediaBuffer *>(photo_buffer_wrapper);
BYTE *photo_buffer = NULL;
HRESULT result = media_buffer->Lock(&photo_buffer, NULL, NULL);
if (SUCCEEDED(result))
{
// do image processing stuff here (with OpenCV) using photo_buffer
media_buffer->Unlock;
}
}
It appears fine to me. It does compile fine too. But at run time, I get an access violation and the applications crashes on Hololens.
Exception Code: 0xC0000005
Exception Information: The thread tried to read from or write to a virtual address for which it does not have the appropriate access.
Anyone sees the problem? Something to do with the way I pass the IMFMediaBuffer object from managed to unmanaged environment?
Thank you very much!
I will answer my own question.
The photo_buffer_wrapper
is not a pointer to IMFMediaBuffer
as I thought but a pointer to IUnknown
. Here is the modified native function that works as intended:
// UNMANAGED ENVIRONMENT
#include <Mfidl.h> // for IMFMediaBuffer support + other COM stuff
void process_photo(void *photo_buffer_unknown, int photo_width, int photo_height)
{
// photo_buffer_unknown = photoCaptureFrame.GetUnsafePointerToBuffer() in managed environment which is an IntPtr
IMFMediaBuffer *media_buffer;
if (SUCCEEDED(reinterpret_cast<IUnknown *>(photo_buffer_unknown)->QueryInterface<IMFMediaBuffer>(&media_buffer)))
{
BYTE* photo_buffer = NULL;
if (SUCCEEDED(media_buffer->Lock(&photo_buffer, NULL, NULL)))
{
// process photo_buffer with OpenCV (wrapped in a cv::Mat)
media_buffer->Unlock();
media_buffer->Release(); // QueryInterface on IUnknown has incremented reference count by one
}
}
}
NB: The pointer returned from photoCaptureFrame.GetUnsafePointerToBuffer()
still has to be released in managed environment like in my question:
// MANAGED ENVIRONMENT
IntPtr mediaBufferUnknownPtr = photoCaptureFrame.GetUnsafePointerToBuffer();
// send mediaBufferUnknownPtr to native function through extern DLL call
Marshal.Release(mediaBufferUnknownPtr)