Given camera matrices, how to find point correspondances using OpenCV?

Question

I'm following this tutorial, which uses Features2D + Homography. If I have known camera matrix for each image, how can I optimize the result? I tried some images, but it didn't work well.

//Edit
After reading some materials, I think I should rectify two image first. But the rectification is not perfect, so a vertical line on image 1 correspond a vertical band on image 2 generally. Are there any good algorithms?

Answer 1

I'm not sure if I understand your problem. You want to find corresponding points between the images or you want to improve the correctness of your matches by use of the camera intrinsics?

In principle, in order to use camera geometry for finding matches, you would need the fundamental or essential matrix, depending on wether you know the camera intrinsics (i.e. calibrated camera). That means, you would need an estimate for the relative rotation and translation of the camera. Then, by computing the epipolar lines corresponding to the features found in one image, you would need to search along those lines in the second image to find the best match. However, I think it would be better to simply rely on automatic feature matching. Given the fundamental/essential matrix, you could try your luck with correctMatches, which will move the correspondences such that the reprojection error is minimised.

Tips for better matches

To increase the stability and saliency of automatic matches, it usually pays to

• Adjust the parameters of the feature detector
• Try different detection algorithms

• Perform a ratio test to filter out those keypoints which have a very similar second-best match and are therefore unstable. This is done like this:

  Mat descriptors_1, descriptors_2; // obtained from feature detector
  BFMatcher matcher;
  vector<DMatch> matches;
  matcher = BFMatcher(NORM_L2, false); // norm depends on feature detector
  vector<vector<DMatch>> match_candidates;
  const float ratio = 0.8; // or something
  matcher.knnMatch(descriptors_1, descriptors_2, match_candidates, 2);
  for (int i = 0; i < match_candidates.size(); i++)
  {
     if (match_candidates[i][0].distance < ratio * match_candidates[i][1].distance)
        matches.push_back(match_candidates[i][0]);
  }
  

• A more involved way of filtering would be to compute the reprojection error for each keypoint in the first frame. This means to compute the corresponding epipolar line in the second image and then checking how far its supposed matching point is away from that line. Throwing away those points whose distance exceeds some threshold would remove the matches which are incompatible with the epiploar geometry (which I assume would be known). Computing the error can be done like this (I honestly do not remember where I took this code from and I may have modified it a bit, also the SO editor is buggy when code is inside lists, sorry for the bad formatting):

  double computeReprojectionError(vector& imgpts1, vector& imgpts2, Mat& inlier_mask, const Mat& F)
  {
  double err = 0;
  vector lines[2];
  int npt = sum(inlier_mask)[0]; 
  
  
  // strip outliers so validation is constrained to the correspondences
  // which were used to estimate F
  vector imgpts1_copy(npt), 
    imgpts2_copy(npt);
  int c = 0;
  for (int k = 0; k < inlier_mask.size().height; k++) 
  {
    if (inlier_mask.at(0,k) == 1) 
    {
       imgpts1_copy[c] = imgpts1[k];
       imgpts2_copy[c] = imgpts2[k];
       c++;
    } 
  }
  
  Mat imgpt[2] = { Mat(imgpts1_copy), Mat(imgpts2_copy) };
  computeCorrespondEpilines(imgpt[0], 1, F, lines[0]);
  computeCorrespondEpilines(imgpt1, 2, F, lines1);
  for(int j = 0; j < npt; j++ )
  {
    // error is computed as the distance between a point u_l = (x,y) and the epipolar line of its corresponding point u_r in the second image plus the reverse, so errij = d(u_l, F^T * u_r) + d(u_r, F*u_l)
    Point2f u_l = imgpts1_copy[j], // for the purpose of this function, we imagine imgpts1 to be the "left" image and imgpts2 the "right" one. Doesn't make a difference
            u_r = imgpts2_copy[j];
    float a2 = lines1[j][0], // epipolar line
          b2 = lines1[j]1,
          c2 = lines1[j][2];
    float norm_factor2 = sqrt(pow(a2, 2) + pow(b2, 2));
    float a1 = lines[0][j][0],
          b1 = lines[0][j]1,
          c1 = lines[0][j][2];
    float norm_factor1 = sqrt(pow(a1, 2) + pow(b1, 2));
  
  double errij =
       fabs(u_l.x * a2 + u_l.y * b2 + c2) / norm_factor2 +
       fabs(u_r.x * a1 + u_r.y * b1 + c1) / norm_factor1; // distance of (x,y) to line (a,b,c) = ax + by + c / (a^2 + b^2)
    err += errij; // at this point, apply threshold and mark bad matches
  }
  
  return err / npt;
  }
  

  The point is, grab the fundamental matrix, use it to compute epilines for all the points and then compute the distance (the lines are given in a parametric form so you need to do some algebra to get the distance).
  This is somewhat similar in outcome to what findFundamentalMat with the RANSAC method does. It returns a mask wherein for each match there is either a 1, meaning that it was used to estimate the matrix, or a 0 if it was thrown out. But estimating the fundamental Matrix like this will probably be less accurate than using chessboards.