How to check if feature descriptors and matches are correct?
I'm trying to find common over laps between two images and for this I am using a ORB feature detector and BEBLID feature descriptor. Using these features, find the homography between them and align the images. The function code is as follows:
for pair in image_pairs:
img1 = cv2.cvtColor(pair[0], cv2.COLOR_BGR2GRAY)
img2 = cv2.cvtColor(pair[1], cv2.COLOR_BGR2GRAY)
detector = cv2.ORB_create(1000)
kpts1 = detector.detect(img1, None)
kpts2 = detector.detect(img2, None)
descriptor = cv2.xfeatures2d.BEBLID_create(0.75)
kpts1, desc1 = descriptor.compute(img1, kpts1)
kpts2, desc2 = descriptor.compute(img2, kpts2)
method = cv2.DESCRIPTOR_MATCHER_BRUTEFORCE_HAMMING
matcher = cv2.DescriptorMatcher_create(method)
matches = matcher.match(desc1, desc2, None)
matches = sorted(matches, key=lambda x: x.distance)
percentage = 0.2
keep = int(len(matches) * percentage)
matches = matches[:keep]
matchedVis = cv2.drawMatches(self.images[pair[0]], kpts1, self.images[pair[1]], kpts2, matches, None)
cv2.imwrite("feature_match.png", matchedVis)
ptsA = np.zeros((len(matches), 2), dtype="float")
ptsB = np.zeros((len(matches), 2), dtype="float")
for (i, m) in enumerate(matches):
ptsA[i] = kpts1[m.queryIdx].pt
ptsB[i] = kpts2[m.trainIdx].pt
(H, mask) = cv2.findHomography(ptsA, ptsB, method=cv2.RANSAC)
(h, w) = img2.shape[:2]
aligned = cv2.warpPerspective(self.images[pair[0]], H, (w, h))
cv2.imwrite("wrap.png", aligned)
A successful alignment of two images looks like:
And an unsuccessful alignment of two images looks like:
Some of the images in image_pairs list have no common overlaps and hence the alignment fails. Is there a way to detect such failures or even detect a successful alignment without explicitly looking at the warp.png images?
This thread and its two top answers are a useful resource for what you are doing:
Detecting garbage homographies from findHomography in OpenCV?
One of the things the selected answer suggests is to check the determinant of the homography matrix. Where negative determinants signals a "flipped image", while a very large or very small determinant signals a blown-up and shrunken results respectively.
I think this is a great way to filter out most of the garbage homography RANSAC happens to give you.
Great supplementary material on determinants of linear transformations
However, in your example case with the image "twisting", I frustratingly cannot find a proper math term for describing the transformation.
I've personally dealt with a simpler problem, where I had 4 control points at the corner of every image I want to transform, but I manually implemented the "find homography" function from scratch. The issue arises when I mismatched the control points like this:
(Pardon my terrible hand-drawn illustration)
Where two of the control points are "swapped" and the transformed image is "twisted". With this illustration, I implemented a naive, brute-force check to verify that no two borders in the transformed image can intersect each other.
I found this Korean blog describing this issue, where they also appear to provide mathematical checks for any "abnormal conversions". According to the blog, checking for negative determinants is enough to catch these twisting transformations. They also provided additional checks for ensuring the image doesn't blow up in size or gets shrunken too much. Here is a Google translated excerpt:
Note that threshold values of sx, sy and P can be adjusted for your usecase.
Finally, the blog mentions that these checks will not catch "concave transformations" (see image in blog) and that you should manually check for them. Google translated text from blog:
However, the case of concave is not inspected under the above D<0 condition. To find the concave case, first transform the four points (0, 0), (1, 0), (0, 1), (1, 0) into a 2×2 submatrix and then check whether the transformed result is concave. should be inspected. To determine whether it is concave or not, refer to [Mathematics] - Calculating the Area (Area) of a Polygonal Figure.