OpenCV + Python: Calculate stereo reprojection error

Question

I want to do something similar to this question, but for stereoCalibrate() instead of calibrateCamera(). That is, compute the reprojection error for a stereo camera calibration.

My reduced example looks like this:

import cv2
import numpy as np

def calibrate_stereo(w, h, objpoints, imgpoints_l, imgpoints_r):
    stereocalib_criteria = (cv2.TERM_CRITERIA_COUNT + cv2.TERM_CRITERIA_EPS , 1000, 1e-6)
    retval, A1, D1, A2, D2, R, T, E, F = cv2.stereoCalibrate(objpoints,imgpoints_l, imgpoints_r,None,None,None,None, (w,h), flags=0, criteria=stereocalib_criteria)

    return (retval, (A1,D1,A2,D2, R, T, E, F))

def calc_rms_stereo(objectpoints, imgpoints_l, imgpoints_r, A1, D1, A2, D2, R, T):
    tot_error = 0
    total_points = 0

    for i, objpoints in enumerate(objectpoints):
        # calculate world <-> cam1 transformation
        _, rvec_l, tvec_l,_ = cv2.solvePnPRansac(objpoints, imgpoints_l[i], A1, D1)

        # compute reprojection error for cam1
        rp_l, _ = cv2.projectPoints(objpoints, rvec_l, tvec_l, A1, D1)
        tot_error += np.sum(np.square(np.float64(imgpoints_l[i] - rp_l)))
        total_points += len(objpoints)

        # calculate world <-> cam2 transformation
        rvec_r, tvec_r  = cv2.composeRT(rvec_l,tvec_l,cv2.Rodrigues(R)[0],T)[:2]

        # compute reprojection error for cam2
        rp_r,_ = cv2.projectPoints(objpoints, rvec_r, tvec_r, A2, D2)
        tot_error += np.square(imgpoints_r[i] - rp_r).sum()
        total_points += len(objpoints)

    mean_error = np.sqrt(tot_error/total_points)

    return mean_error


if __name__ == "__main__":    
    # omitted: reading values for w,h, objectPoints, imgpoints_l, imgpoints_r from file (format as expected by the OpenCV functions)
    # [...]

    rms, (A1,D1,A2,D2,R,T,_,_) = calibrate_stereo(w, h, objectpoints, imgpoints_l, imgpoints_r)

    print("RMS (stereo calib): {}".format(rms))

    rms_2 = calc_rms_stereo(objectpoints, imgpoints_l, imgpoints_r, A1, D1, A2, D2, R, T)    
    print("RMS (custom calculation):", rms_2)

Sample output:

RMS (stereo calib): 0.14342257926694932
RMS (custom calculation): 0.356273345751

As far as I can tell, the computation in the source code of stereoCalibrate() is very similar to mine. What am I missing?

OpenCV 3.3.0 on Ubuntu

Answer 1

I solved it after implementing a custom stereo calibration algorithm based on the OpenCV implementation.

The difference between the reprojection error calculated inside cv2.stereoCalibrate() and my custom calculation stems from different values for the extrinsic parameters rvec_l and tvec_l. These vectors describe the rotation and translation between the left camera and the calibration pattern for each image. cv2.solvePnpRansac() yields optimized values based only on the reprojection error of the left image, while in cv2.stereoCalibrate() those values are optimized together with R and T based on the reprojection error in both images of each stereo pair.

If one wants to exactly replicate the RMS value that is returned by cv2.stereoCalibrate(), one has to modify the C/C++ source code of cv::stereoCalibrate() to return the optimized extrinsic parameters as well (cv::calibrateCamera() does that already for monocular calibration).