Search code examples
pythonopencvimage-processingface-detectiondlib

how to extract self-defined ROI with dlib facelandmarks?

enter image description here

I don't know how to extract the irregular area surrounded by green lines. i.e., the left cheek and the right cheek of a face.

from collections import OrderedDict
import numpy as np
import cv2
import dlib
import imutils

CHEEK_IDXS = OrderedDict([("left_cheek", (1, 2, 3, 4, 5, 48, 31)),
                          ("right_cheek", (11, 12, 13, 14, 15, 35, 54))
                          ])

detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor("shape_predictor_68_face_landmarks.dat")

img = cv2.imread('Tom_Cruise.jpg')
img = imutils.resize(img, width=600)

overlay = img.copy()
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

detections = detector(gray, 0)
for k, d in enumerate(detections):
    shape = predictor(gray, d)
    for (_, name) in enumerate(CHEEK_IDXS.keys()):
        pts = np.zeros((len(CHEEK_IDXS[name]), 2), np.int32)
        for i, j in enumerate(CHEEK_IDXS[name]):
            pts[i] = [shape.part(j).x, shape.part(j).y]

        pts = pts.reshape((-1, 1, 2))
        cv2.polylines(overlay, [pts], True, (0, 255, 0), thickness=2)

    cv2.imshow("Image", overlay)

    cv2.waitKey(0)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
cv2.destroyAllWindows()

I know if just simply extract a rectangular area from the face as cheeks, the code can be like this

ROI1 = img[shape[29][1]:shape[33][1], shape[54][0]:shape[12][0]] #right cheeks
ROI1 = img[shape[29][1]:shape[33][1], shape[4][0]:shape[48][0]] #left cheek

enter image description here

but I want to extract the irregular area for subsequent processing, how can i do it ?

Solution

  • You can accomplish this by two simple steps:

    1. Create a mask using the point coordinates you have
    2. Execute bitwise_and operation (crop)

    Code:

    cv2.drawContours(mask, [pts], -1, (255, 255, 255), -1, cv2.LINE_AA)
output = cv2.bitwise_and(img, img, mask=mask)

    Output: enter image description here

    Additionally, if you want to focus on the cropped polygons, you can create a bounding rectangle to the polygons then crop from the output frame like tihs:

    # Create a bounding rects list at global level
bounding_rects = []

# Calculate Bounding Rects for each pts array inside the for loop
bounding_rects.append(cv2.boundingRect(pts))

# Assign geometrical values to variables to crop (Use a range(len(bounding_boxes)) for loop here)
enter code here
x1,y1,w1,h1 = bounding_rects[0]
x2,y2,w2,h2, = bounding_rects[1]

# At the end of the program, crop the bounding boxes from output
cropped1= output[y1:y1+h1, x1:x1+w1]
cropped2= output[y2:y2+h2, x2:x2+w2]

    Output:

    enter image description here