rotate a image on the nose and eye axis with opencv

Question

I have the following image with eyes and noise points. The guy in the picture has a minimal tilt and I want to normalize this. I have mark the landmarks with dlib and so I can calculate the tilt.

Now, how can I rotate simply the whole image, that the guy is straight?

LEFT_EYE: [(274, 269), (286, 259), (302, 258), (317, 268), (302, 271), (286, 272)]
RIGHT_EYE : [(388, 264), (401, 254), (417, 252), (431, 261), (419, 265), (403, 265)]
NOSE: [(352, 257), (353, 278), (354, 297), (354, 319)]

As an idea, the nose points on the x axis are 352, 353, 354, 354. Theoretically if I make a matrix transformation, that change the all x points to 352, the noise will be straight in a line.

I think it can be done with a matrix transformation, and the noise or eye points as vector transformation. But I need an approach, how can this be solved.

Answer 1

First, do a linear fit to the nose points, and use the slope to find the angle. To do this, switch x- and y-values so that straight up is an angle of 0 degrees (also, so you get a better fit when the nose is almost straight up to start).

# Nose points
nose =  np.array([(352, 257), (353, 278), (354, 297), (354, 319)])

# Linear fit with x- and y-values switched
m,b = np.polyfit(nose[:,1], nose[:,0], 1)

# Angle in radians
angle = np.arctan(m)

Next, you can use opencv to get a rotation matrix. If the center of rotation doesn't matter, you can just rotate around the center of the image. But below I rotate it around the top of the nose so the points sit vertically along the x=352 line if you do want that. Finally, use the rotation matrix to rotate the image with cv2.warpAffine.

# Get rotation matrix
rot_mat = cv2.getRotationMatrix2D(tuple(nose[0]), -angle*180./np.pi, 1)

# Rotate image
dims = tuple(img.shape[:2][::-1])
rotated_image = cv2.warpAffine(img, rot_mat, dims)

Here's the resulting image: