Search code examples
c#androidopencvblockingcollection

Performance degradation over time with BlockingCollections, Tasks, and OpenCV


I'm writing a simple optical measurement application using Xamarin for Android and the OpenCV C# bindings library.

In an effort to separate the frame grabber from the processing, I've created some blocking collections to pass around raw, and then processed imagery between different threads. I have an issue where over the period of about 30 seconds, the GUI shows beautifully smooth processed video (15s) down to choppy video (10s), then a crash.

The code below shows the definition of the collections. OnCameraFrame (bottom of the code) shoves each new frame into camframes collection. In OnCreate, I run a task called CamProcessor that takes the frame, does many things, and stuffs it into outframes collection. OnCameraFrame then takes that processed frame and shows it to the GUI. For the purposes of this post and testing, I've completely commented out all my processing, so this issue exists simply by passing raw data through the collections.

One other note is that my collections seem to be running very fast. At no point do I ever have more than 1 frame in there, so it's not an overflow issue (I think).

Can anyone point to why this strategy isn't working well?

BlockingCollection<Mat> camframes = new BlockingCollection<Mat>(10);
BlockingCollection<Mat> outframes = new BlockingCollection<Mat>(10);
public CameraBridgeViewBase mOpenCvCameraView { get; private set; }

    protected override void OnCreate(Bundle savedInstanceState)
    {
        //LayoutStuff

        mOpenCvCameraView = FindViewById<CameraBridgeViewBase>(Resource.Id.squish_cam);

        Task.Run(() => camProcessor());
    }

    public void camProcessor()
    {
        while (!camframes.IsCompleted)
        {
            Mat frame = new Mat();
            try
            {
                frame = camframes.Take();
            }
            catch (InvalidOperationException) { }

            Mat frameT = frame.T();
            Core.Flip(frame.T(), frameT, 1);
            Imgproc.Resize(frameT, frameT, frame.Size());
            outframes.Add(frameT);
         }
     }

    public Mat OnCameraFrame(CameraBridgeViewBase.ICvCameraViewFrame inputFrame)
    {
        mRgba = inputFrame.Rgba();
        Mat frame = new Mat();
        Task.Run(() => camframes.Add(mRgba));
        try
        {
            frame = outframes.Take();
        }
        catch (InvalidOperationException) { }

        return frame;
    }

Solution

  • After looking into the Android SDK monitor.bat output, I discovered this was a memory leak. Turns out it's common to the Java openCV wrapper, and is a result of OpenCV's mat heap being much larger than C# expects it to be, so it's not getting garbage collected.

    The solution was to append these at every frame grab:

                 GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
                 GC.Collect();
                 GC.WaitForPendingFinalizers();