Find contours in images with complex background and rich texture in OpenCV 3.3

Question

I'm working on a project to process marble slab images like the one below by using OpenCV 3.3.

Several more samples with different marble textures and sizes I'm working on can be found on https://1drv.ms/f/s!AjoScZ1lKToFheM6wmamv45R7zHwaQ

The requirements are:

1. Separate the marble slab from the background and remove the background (fill with white color) so only the slab is shown.
2. Calculate the area of the slab (the distance from the camera to the marble slab and the parameters of the lens are known)

The strategy I am using is: 1) find contour of the marble slab, 2) remove parts not within the contour, 3) get the area size of the contour, 4) calculate its physical area.

The contour of the slab is shown in the picture below in red (this one was done by hand).

I tried several ways to find contours of the slab in the image but failed to achieve a satisfying result because of the complex background and the rich texture of the marble.

Processing logic I'm using is: convert the image to gray and blur it and use Canny to find edges, then use findContours to find contours, the following is the code:

Mat img = imread('test2.jpg', 1);
Mat gray, edge;
cvtColor(img, gray, COLOR_BGR2GRAY);
blur(gray, gray, Size(3, 3));

Canny(gray, edge, 50, 200, 3);

vector<vector<Point> > contours;
vector<Vec4i> lines;
findContours(edge, contours, RETR_LIST, CHAIN_APPROX_SIMPLE);

cout << "Number of contours detected: " << contours.size() << endl;

vector<Vec4i> hierarchy;

for (int i = 0; i < contours.size(); i++)
{
    drawContours(img, contours, i, Scalar(0, 0, 255), 1, 8, hierarchy, 0, Point());
}

imshow("Output", img);

I tried to tweak the blur and Canny parameters for dozens of combinations and still failed. I also tried to use HoughLinesP to find the edge of the slab with several groups of different parameters but alsofailed to do so.

I'm a newbie to computer vision, the questions I have now are:

1. Am I going towards a wrong way or strategy to find the slab contour? Could there be any better algorithms or combinations? Or do I need to focus on tweaking parameters of Canny/findContours/HoughLinesP algorithms?
2. Is this kind of image really difficult to process due to the complex background?

I'm open to any suggestions that may help me finish my goal. Thank you in advance.

Answer 1

Techniques you can consider

1. Template matching, you may need to prepare a lot of template for different marble(light condition, rotation etc)
2. Train a classifier + region proposals, only adopt this solution if other solutions failed(this solution maybe is the most robust one, but also the most verbose one to implement)

Since you only got 10~20 types of marble slab, I think solution 1 is a good start.

1. Manually find out 4 corners points of the marble, do perspective transform

   pair<Mat, vector<Point2f>> get_target_marble(Mat const &input, vector<Point2f> const &src_pts)
   {
   using namespace cv;
   using namespace std;
   
   Point2f const tl = src_pts[0];
   Point2f const tr = src_pts[1];
   Point2f const br = src_pts[2];
   Point2f const bl = src_pts[3];
   
   auto const euclidean_dist = [](Point const &a, Point const &b)
   {
       return std::sqrt(std::pow(a.x-b.x, 2) + std::pow(a.y - b.y, 2));
   };
   int const max_width = static_cast<int>(std::max(euclidean_dist(br, bl), euclidean_dist(tr, tl)));
   int const max_height = static_cast<int>(std::max(euclidean_dist(tr, br), euclidean_dist(tl, bl)));
   
   vector<Point2f> const src{tl, tr, br, bl};
   vector<Point2f> dst{Point(0,0), Point(max_width -1,0), Point(max_width-1,max_height-1), Point(0,max_height-1)};
   Mat const hmat = getPerspectiveTransform(src, dst);
   Mat target;
   warpPerspective(input, target, hmat, {max_width, max_height});
   
   return std::make_pair(std::move(target), std::move(dst));
   

   }

2. Find out homography matrix between query image(marble slab) and train image(the image may contain marble slab)

   Mat find_homography(Mat const &train, Mat const &query)
   {
   Ptr<AKAZE> akaze = AKAZE::create();
   vector<KeyPoint> query_kpts, train_kpts;
   cv::Mat query_desc, train_desc;
   akaze->detectAndCompute(train, cv::noArray(), query_kpts, query_desc);
   akaze->detectAndCompute(query, cv::noArray(), train_kpts, train_desc);
   
   BFMatcher matcher(NORM_HAMMING);
   vector<vector<DMatch>> nn_matches;
   //top 2 matches because we need to apply David Lowe's ratio test
   matcher.knnMatch(train_desc, query_desc, nn_matches, 2);
   
   vector<KeyPoint> matches1, matches2;
   for(auto const &m : nn_matches){
       float const dist1 = m[0].distance;
       float const dist2 = m[1].distance;
       if(dist1 < 0.7 * dist2){
           matches1.emplace_back(train_kpts[m[0].queryIdx]);
           matches2.emplace_back(query_kpts[m[0].trainIdx]);
       }
   }
   
   if(matches1.size() > 4){
       std::vector<cv::Point2f> points1, points2;
       for(size_t i = 0; i != matches1.size(); ++i){
           points1.emplace_back(matches1[i].pt);
           points2.emplace_back(matches2[i].pt);
       }
       return cv::findHomography(points1, points2, cv::RANSAC, 5.0);
   }
   
   return {};
   

   }

3. Map the 4 lines of query image to target image

   vector<Point2f> query_points;
   vector<Point> qpts;
   perspectiveTransform(dst, query_points, hmat);
   for(auto const &pt : query_points){
       cout<<pt<<endl;
       qpts.emplace_back(pt);
   }
   
   polylines(input, qpts, true, {255,0,0}, 2);
   

You will need to prepare 10~20 images for this solution, prefer the one exist most of the matches points to locate your marble slab. If performance is an issue, decrease resolution of the image, you do not need a big images to get results.

Complete codes are put at github.

ps : I do not know the details of your project, if there are only 10~20 types of marble slab + all of them have good features to track, you do not need 3 months to solve it(But you can tell your bosses/customers you need 3 months :), sometimes better performance only lead to more chores but not more money).