Using background threads for image processing in C++ in unity leads to a crash

Question

I am trying to do some image processing in Unity and OpenCV in the background because it needs around 60ms-100ms to complete and slows down the application drastically when executed on the main thread.

I do this inside a "camera frame available" callback which executes 30 times a second. I am using "Task Parallel" from the asset store.

C# Code

private const int numMaximumThreads = 3;
private volatile int currentNumberOfWorkingThreads = 0;

unsafe private void OnRawVideoFrameAvailableYUV(MLCameraResultExtras resultExtras, YUVFrameInfo frameInfo, MLCameraFrameMetadata frameMetadata)
{
    //only have a maximum of 3 threads
    if(currentNumberOfWorkingThreads < numMaximumThreads)
    {
        //run on background
        UnityTask.Run(() =>
        {
            currentNumberOfWorkingThreads++;
            MLCamera.GetFramePose(resultExtras.VcamTimestampUs * 1000, out translationMatrix);

            float x = 0, y = 0, z = 0, rx = 0, ry = 0, rz = 0;
            //fix the bytearray for OpenCV on c++ side
            fixed (byte* yBuffer = frameInfo.Y.Data)//camera uses YUV instead of RGB
            {
                fixed (byte* uBuffer = frameInfo.U.Data)
                {
                    fixed (byte* vBuffer = frameInfo.V.Data)
                    {   //call OpenCV / c++ function
                        Interop.Detect((IntPtr)yBuffer, (IntPtr)uBuffer, (IntPtr)vBuffer, camWidth, camHeight, ref x, ref y, ref z, ref rx, ref ry, ref rz);
                    }
                }
            }
            //r to rz are translation and rotation vectors that OpenCV returns
            return new float[6] { x, y, z, rx, ry, rz };
        //after completion, move gameobjects on unity main thread
        }).ContinueOnUIThread((r) =>
        {
            if (!(r.Result[0] == 0 && r.Result[3] == 0))//very probably not detected when x and rx are 0
            {
                Utils.UpdatePosition(r.Result[0], r.Result[1], r.Result[2]);
                Utils.UpdateRotation(r.Result[3], r.Result[4], r.Result[5]);
                Utils.PlaceArucoObject(markerCube, virtualCamera, translationMatrix);
            }
            currentNumberOfWorkingThreads--;
        });
    }
}

My Interop function is as follows:

C# Code

[DllImport("ml_aruco_api")]
internal static extern void Detect(IntPtr yBuffer, IntPtr uBuffer, IntPtr vBuffer, int width, int height, ref float x, ref float y, ref float z, ref float rx, ref float ry, ref float rz);

And finally, the OpenCV C++ side (shortened)

extern "C" void Detect(unsigned char* yBuffer, unsigned char* uBuffer, unsigned char* vBuffer, int width, int height, float& x, float& y, float& z, float& rx, float& ry, float& rz) {
    //(reconstruct image and detect markers on it...)

    if (detected) {
        x = m4.Tvec.at<float>(0, 0);
        y = m4.Tvec.at<float>(0, 1);
        z = m4.Tvec.at<float>(0, 2);
        rx = m4.Rvec.at<float>(0, 0);
        ry = m4.Rvec.at<float>(0, 1);
        rz = m4.Rvec.at<float>(0, 2);
    }
    else {
        x = y = z = rx = ry = rz = 0;
    }
}

This all works for a few seconds, then crashes. And it doesn't run at 60fps but fluctuates strongly around 15-30fps. I don't know much about memory handling between C# and C++ so that's probably the problem here. I also don't understand why the application runs so slowly when it's supposedly run on a background thread. I also tried Unity's job system to no avail, it was working without crashing but slowly as well.

Could someone point me to the right direction?

Edit: I limited the number of maximum threads to 1 and it stopped the crashing. The FPS are now around 30-40 and I wonder why it's not on 60 fps. Commenting out the Interop.Detect() function gives me 60. Do I have to thread the OpenCV function itself?

Answer 1

Use Span and Memory in .net to share data between managed or unmanaged

ref :

https://learn.microsoft.com/en-us/dotnet/standard/memory-and-spans/memory-t-usage-guidelines

https://medium.com/@antao.almada/p-invoking-using-span-t-a398b86f95d3