OpenCV Shape detection ignoring marks along edges using C#

Question

I'm trying to pull out a bounding rect on a page and have to account for the users (small children) who may write on the page messing with outline boxes.

Like this: Top left of the box has writing which interferes with the bounding box contours

When attempting to get the contours and pull the box it seems the writings that are on the boxes edge cause the contours to no longer have 4 points or may also just extend the box in a direction: Writing on the upper left altered how my box is grabbed

Below is my code for how I get the contours of the whole page after performing gray, threshold, and Gaussian on the original Image.

//Fining the Question circles
Mat copy = guassianBlurImage.Clone();
var circles = Cv2.HoughCircles(copy, HoughModes.Gradient, 1, 10, 1, 25, 13, 14);

//Whitling down the possible selections & Find Bounding Boxes + Biggest Contour Box
OpenCvSharp.Point[][] contours = null;
HierarchyIndex[] hierachy = null;

//External because list or tree will also get a inner rectangle for the same box
Cv2.FindContours(thresholdImage, out contours, out hierachy, RetrievalModes.External, ContourApproximationModes.ApproxNone);

List<OpenCvSharp.Rect> boundingBoxes = new List<OpenCvSharp.Rect>();
List<OpenCvSharp.Rect> questionBoxes = new List<OpenCvSharp.Rect>();
Dictionary<string, Point2f[]> questionBoxContourDic = new Dictionary<string, Point2f[]>();
   foreach (OpenCvSharp.Point[] c in contours)
   {
       var area = Cv2.ContourArea(c);
       var peri = Cv2.ArcLength(c, true);
       var approx = Cv2.ApproxPolyDP(c, 0.02 * peri, true);
       OpenCvSharp.Rect r = Cv2.BoundingRect(c);

       //Check if any question bubbles exist in the rect
       bool containsQuestionBubble = false;
       foreach (var cir in circles)
       {
           OpenCvSharp.Point circleCenter = new OpenCvSharp.Point(cir.Center.X, cir.Center.Y);
           if(r.Contains(circleCenter))
           {
               containsQuestionBubble = true;
               break;
           }
       }

       //Ignore words and small boxes
       if (approx.Length == 4 && area > 500)
       {
           Mat temp = new Mat();
           var xCenter = (r.TopLeft.X + r.BottomRight.X) / 2;
           var yCenter = (r.TopLeft.Y + r.BottomRight.Y) / 2;
           Cv2.GetRectSubPix(thresholdImage, new OpenCvSharp.Size(r.Width, r.Height), new Point2f(xCenter, yCenter), temp);

           int total = Cv2.CountNonZero(temp);
           //Adds the Bounding boxes to a list for later
           if (total > 4000 && total < 4700)
           {
               boundingBoxes.Add(r);
           }
           //Adds the questionbox to a list for later
           if (containsQuestionBubble)
           {
               int i = questionBoxContourDic.Count() + 1;
               questionBoxes.Add(r);
               RotatedRect rr = Cv2.MinAreaRect(c);
               Point2f[] corners = Cv2.BoxPoints(rr);
               //Question box name with corner points
               questionBoxContourDic.Add("QuestionBox" + i, corners);
           }
       }

If possible I would also like to avoid having the user define areas of interest since this process should remain self sufficient.

Answer 1

I ended up just creating multiple masks for things like bounding boxes, question boxes, etc... and saving them to a json file. The user will have to define these general areas before running the program but since these documents hardly change its not too big of a deal plus they only have to do it once. This allows me to pull up a mask when needed to check if my desired box is within the bounds of the masks white area.