Perspective N point solution: Position drifting linearly with rotation angles?

Question

I am using a P3P algorithm to compute the rotation and translation of a camera given pre-mapped 3D points, and their corresponding projections on the 2D plane (PNP problem). The algorithm I am using is the one described in a CVPR 2017 paper, with implementations in both OpenCV and OpenMVG (I am using the latter). Naturally, the algorithm works inside a RANSAC framework.

Strangely, I am noticing a precise 'drift' in the position computed, that changes with the rotation angles. I.e.: If I hold the camera's position constant and just rotate it, according to the OpenCV coordinate convention, changes in pitch cause the position to drift in the Y direction, and changes in yaw cause the position to drift in the X direction (I have not tested roll yet.) When I tried to fit a curve to this dataset of pitch vs Y and yaw vs X values, I noticed a pretty constant variation. After removing scale factors and converting the angles into radians, this is the profile I see:

X ~= -4.0 * yaw_angle
Y ~= 4.0 * pitch_angle

(Translation units in meters with a scale factor of 1.0, rotation angles in radians)

Is this some sort of a commonly known transformation I am failing to account for? It's linear, so I am assuming it cannot be dependent on the rotation angle. I have skimmed through the paper and some relevant resources but I am unable to figure out the cause of this relationship.

Curve fitting results from MATLAB:

Answer 1

The problem was a bug in my code where I was accessing the translation part of the solution directly, and not the camera position, so it was indeed a missing transformation.

For future reference, the true camera position from a PNP solution needs to be computed as

C = -R' * t

R is the rotation matrix and t is the final translation vector returned by the PNP algorithm. Being a linear operator, this encodes a purely linear relationship with respect to the rotation angles.