Is there a way to smooth out face landmarks from face_recognition? Maybe via PIL?

Question

I am trying to make an eye-replacement program that can fill eyes with an image. For finding the eyes, I am using face_recognition by Ageitgey. However, the eye detection comes out very jagged.

(I'm not talking about anti-aliasing, btw. I'll use super-sampling to solve that later)

Here's my tiny bit of code:

from PIL import Image, ImageDraw
import face_recognition

image = face_recognition.load_image_file("media/test_input_image.jpg")

face_landmark_list = face_recognition.face_landmarks(image)

for face_landmarks in face_landmark_list:
    pil_image = Image.fromarray(image)
    d = ImageDraw.Draw(pil_image, 'RGBA')

    d.polygon(face_landmarks["right_eye"], fill=(255, 0, 0, 255))

    pil_image.show()

example: [Daniel's surprisingly terrifying badly selected eyes]

I want it looking more smooth. I am looking to achieve something like the green eye on the left, but am currently getting the red eye on the right. (The green eye was drawn on with Gimp.)

So, basically, is there a way to go from the red result, to the green?

Answer 1

Method 1: (Easy)

• Use quadratic or cubic regression to fit a curve using left/right and 2 upper points
• Do the same thing for left/right and 2 lower points.
• Sampling depends on how many points you want for each.

Sample python code:

import numpy.polynomial.polynomial as poly
import numpy as np
import matplotlib.pyplot as plt

# Grab x, y coordinates from eyes in face_recognition result
# Here is the example point.
x = np.array([1, 2, 4, 5])
y = np.array([1, 1.5, 1.5, 1])

# x_new: upsampling 40 points from x
x_new = np.linspace(x[0], x[-1], num=len(x)*10)

coefs = poly.polyfit(x, y, 3)
y_new = poly.polyval(x_new, coefs)

plt.plot(x_new, y_new,'r-')
plt.plot(x,y,'o')
plt.show()

Method 2: (Difficult)

• Use dlib, re-train object detector to just detect eyes with more than 6 points, e.g. 64 points for just one eye, you'll get smoother result.