What does MATLAB's estimateUncalibratedRectification do expressed mathematically ? What does it calculate exactly?

Question

What does MATLAB's estimateUncalibratedRectification do in mathematical/geometrical terms ?

What does it calculate exactly ?

As far as I understand, if the camera parameters are unknown then only the fundamental matrix can be computed from two images, not the essential matrix.

So, as far as I understand, estimateUncalibratedRectification's result should be ambiguous up to some kind of a transformation T because the fundamental matrix - that can be computed from two images if the camera's intrinsic parameters are not known - is ambiguous up to an arbitrary projective transformation.

Is this correct ?

My guess is that estimateUncalibratedRectification computes a projective transformation P1 for image1 and another projective transformation P2 for image2 in such a way that when using these two transformations (P1 and P2) on the corresponding images then the resulting images (R1 and R2) will be rectified in the sense that the corresponding epipolar lines will be in the same rows, as shown in the image below.

My question is: how ambiguous is this result ? My feeling is that the resulting transformations P1 and P2 are ambiguous up to some transformation T but I don't really see what this T can be.

Could someone please explain how estimateUncalibratedRectification works conceptually/mathematically/geometrically and also tell what T can be ?

In other words, what is the transformation T which when applied to R1 and R2 will result in an image pair TR1 and TR2 which will have the same rectified property as R1 and R2, namely that corresponding epipolar lines appear in matching rows in TR1 and TR2, just like they do in R1 and R2?

What is this T ? Is there such a T ?

PS.: I have read the code of estimateUncalibratedRectification.m before posting this question but it made me no wiser.

Answer 1

If the intrinsics are not known, the result is ambiguous up to a projective transformation. In other words, if you use estimateUncalibratedRectification to rectify a pair of images, and then compute disparity and do the 3D reconstruction, then you will reconstruct the 3D scene up to a projective transformation. Straight lines will be straight, parallel lines will be parallel, but your angles and sizes will likely be wrong.

To determine what that projective transformation is, you would need more information. If you know the camera intrinsics, then you get up to scale reconstruction. In other words you get the correct angles and relative sizes. To get the correct scale you would either need to know the baseline (distance between the cameras) or the size of some reference object in the scene.

The more straight-forward way to approach this is to calibrate your cameras using the Camera Calibrator app or the Stereo Camera Calibrator app.