Dlib not detecting face in kurento opencv filter

Question

I have created an opencv filter that can detect if a person blinks for Kurento the WebRTC framework. My code works in a standalone opencv app. However, once I converted to the opencv filter for Kurento it started playing up. When the module/filter was compiled without optimisation flags it would briefly detect the face and draw contours around the eyes. However, after compiling the module/filter with optimisation flags, performance improved, but no face was being detected. Here's the code I have in the filter:

 void BlinkDetectorOpenCVImpl::process(cv::Mat &mat) {

            std::vector <dlib::rectangle> faces;

            // Just resize input image if you want
            resize(mat, mat, Size(800, 450));

            cv_image <rgb_alpha_pixel> cimg(mat);
            dlib::array2d<unsigned char> img_gray;
            dlib::assign_image(img_gray, cimg);
            faces = detector(img_gray);
            std::cout << "XXXXXXXXXXXXXXXXXXXXX FACES: " << faces.size() << std::endl;
            std::vector <full_object_detection> shapes;
            for (unsigned long i = 0; i < faces.size(); ++i) {
                full_object_detection shape = pose_model(cimg, faces[i]);
                std::vector <Point> left_eye_points = get_points_for_eye(shape, LEFT_EYE_START, LEFT_EYE_END);
                std::vector <Point> right_eye_points = get_points_for_eye(shape, RIGHT_EYE_START, RIGHT_EYE_END);
                double left_eye_ear = get_eye_aspect_ratio(left_eye_points);
                double right_eye_ear = get_eye_aspect_ratio(right_eye_points);

                double ear = (left_eye_ear + right_eye_ear) / 2.0;

                // Draw left eye
                std::vector <std::vector<Point>> contours;
                contours.push_back(left_eye_points);
                std::vector <std::vector<Point>> hull(1);
                convexHull(contours[0], hull[0]);
                drawContours(mat, hull, -1, Scalar(0, 255, 0));

                // Draw right eye
                contours[0] = right_eye_points;
                convexHull(contours[0], hull[0]);
                drawContours(mat, hull, -1, Scalar(0, 255, 0));

                if (ear < EYE_AR_THRESH) {
                    counter++;
                } else {
                    if (counter >= EYE_AR_CONSEC_FRAMES) {
                        total++;
                        /*  std::string sJson = "{\"blink\": \"blink\"}";

                          try
                          {
                              onResult event(getSharedFromThis(), onResult::getName(), sJson);
                              signalonResult(event);
                          }
                          catch (std::bad_weak_ptr &e)
                          {
                          }*/
                    }

                    counter = 0;
                }

                cv::putText(mat, (boost::format{"Blinks: %d"} % total).str(), cv::Point(10, 30),
                            cv::FONT_HERSHEY_SIMPLEX,
                            0.7, Scalar(0, 0, 255), 2);
                cv::putText(mat, (boost::format{"EAR: %.2f"} % ear).str(), cv::Point(300, 30),
                            cv::FONT_HERSHEY_SIMPLEX,
                            0.7, Scalar(0, 0, 255), 2);
            }
        }


    } /* blinkdetector */

Answer 1

I was able to fix my own problem. I found that instead of resizing the image to an arbitrary resolution you should resize it by half the width and half the height of the actual image resolution. Resizing an image to a lower size makes Dlib face detection fast. So here's what I did to solve the issue:

    Mat tmpMat = mat.clone();
    resize(tmpMat, tmpMat, Size(tmpMat.size().width / 2, tmpMat.size().height / 2));

I had to clone the image sent by Kurento to my method because for some odd reason the original Mat doesn't show the contours when turned into a Dlib image with cv_image.