opencv - Detecting a rectangle on a photo of paper with inner elements

Question

So I am trying to read the image below.

I have been able to make an adaptive threshold and detect a rotation angle (I am not sure if I had to rotate the image)

What I am struggling with is detecting the rectangle that includes the form. I tried different approaches such as opencv's findContours(). The biggest contour that it is able to find is a box with first name.

After that I decided to use HoughLinesP, but it finds a lot of lines and I do not know how can I filter them. It would also be handy to detect the rectangle to deskew the form and after that I will be able to read the answers easily. So I am already thinking of adding black square markers to the corners.. But maybe someone can give me some ideas of how to do it right.

HoughLinesP (I use nodejs, but I can read python and c++):

const imageSize = {
    width: gray.cols,
    height: gray.rows
  };

  const threshold_min = 200;
  const ratio_min_max = 1;
  const edges = gray.canny(threshold_min,threshold_min*ratio_min_max,3);

  const minLineLength = imageSize.width / 4, 
        maxLineGap = 10, 
        threshold = 100;
  const lines = edges.houghLinesP(1, Math.PI/180, threshold, minLineLength, maxLineGap);

  //draw lines on the output
  for( let i = 0; i < lines.length; i++ )  {  
    const l = lines[i];  
    const {x,y,z,w} = l;
    output.drawLine(
      cv.Point(w, x), 
      cv.Point(y, z), 
      new cv.Vec(Math.random()*255,Math.random()*255,Math.random()*255), 
      // new cv.Vec(0,0,255), 
      2, 
      // 1
    );   
  }
  //end draw lines

Answer 1

Ok, so I was able to detect the rectangle by using dilation:

  let {area, contour} = getMaxContour(gray);

  let dilateIterations = 0;

  const MAX_DILATE_ITERATIONS = 9;
  while(area<MINIMAL_POSSIBLE_AREA && dilateIterations<MAX_DILATE_ITERATIONS){
    dilateIterations++;
    gray = gray.dilate(new cv.Mat(), new cv.Point(-1,-1), 1, cv.BORDER_CONSTANT);

    let result = getMaxContour(gray);
    contour = result.contour;
    area = result.area;

    if(DEBUG) {
      writeImage(`dilated_${dilateIterations}.png`, gray);
    }
  }

And max contour code:

const getMaxContour = (image) => {
  const contours = image.findContours(cv.RETR_LIST,cv.CHAIN_APPROX_SIMPLE);
  let maxAreaFound = 0;
  let maxContour = [];
  let contourObj = null;
  console.log(`Found ${contours.length} contours.`);
  contours.forEach((contour,i)=>{
    // const perimeter = cv.arcLength(contour, true);
    const perimeter = contour.arcLength(true);
    const approx = contour.approxPolyDP(0.1*perimeter, true);
    const area = contour.moments()['m00'];

    if (approx.length == 4 && maxAreaFound<area){
      maxAreaFound = area;        
      maxContour = approx;
      contourObj=contour;
    }  
  });

  console.log(JSON.stringify(contourObj))
  console.log(`Max contour area found ${maxAreaFound}.`);

  return {
    contour:maxContour,
    area:maxAreaFound
  };
}

After that I was able to highlight corners and do further perspective fix.