Detecting/Segmenting Face and Facial Features WITHOUT using a trained model: openCV C++

Question

Using a single image -I want to know if it is possible to detect a face(object) and facial landmarks (eyes, mouth, nose) in a image without using some pretrained model such as "haarcascade_eye" or "lbpcascade_frontalface" or anything related. I want to process the face based on no user input like manually selecting a region of interest (ROI) using selectROI(). I've tried to segment the image based on openCV tutorials, but I'm struggling with separating the background and body from the head. I've also tried to use algorithms such as SIFT/SURF/FAST, but the algorithm detects random key points and is not useful later if I wish to detect facial landmarks. Does anybody have an example, tutorial, thoughts or ideas they would be willing to share?

The image I am using

Answer 1

I would like to point out that when you run SIFT over the image, it will show you all the 'points of interest' in that image. You need to have a base image to map it to, for example, keep a cropped image of an eye, or whatever facial landmark you are focusing on.

Your code will have to extract keypoints from both the 'train' and test images. Once your do that, you can use a matching function such as the BruteForce Matcher or FLANN. After that define your threshold for a good match and you'll be good to go.

Example:

kp_train,des_train = sift.detectAndCompute(image_train,None)
kp_test,des_test = sift.detectAndCompute(image_test,None)


bf = cv2.BFMatcher()
matches = bf.knnMatch(des_train,des_test,k=2)
good = []
for m,n in matches[index]:
    if m.distance < dist_ratio*n.distance:
        good.append([m])

if len(good)>MIN_MATCH_COUNT:
img3 = cv2.drawMatchesKnn(image_train,keypoints_train,image_test,keypoints_test,good,None, flags=2)
cv2.imwrite("Path/To/Save",img3)

Change the value of MIN_MATCH_COUNT according to your application

I would suggest your add a homography function and use RANSAC to get rid of outliers. With this you can even draw a box around your ROI and extract it.

Alternate approach: You can try out a template matching algorithm such as MSTM. This algorithm will take a base image and tile it across the test image and check if it gets a match anywhere. If all your images will be similar to the one you have attached, MSTM can be helpful

P.s It's pretty evident that I'm using Python, you can find the equivalent C++ functions on the opencv documentations

Check this as an example