How to get these ROIs for these images in OpenCV

Question

I have some sample images as follows

what i am trying to do is remove the labels from the image, So the resulting image should look as below

Finally i want to get the rectangles as shown in image

Till now i have code that takes the template and removes the border to get the first result

import cv2
import numpy as np


def remove_templates(image):
    templates = ['images/sample1.jpeg', 'images/sample2.jpeg']
    for template in templates:
        template = cv2.imread(template)
        h, w, _ = template.shape
        res = cv2.matchTemplate(cv2.cvtColor(img, cv2.COLOR_BGR2GRAY), cv2.cvtColor(template, cv2.COLOR_BGR2GRAY), cv2.TM_CCOEFF)
        min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
        top_left = max_loc
        bottom_right = (top_left[0] + w, top_left[1] + h)
        cv2.rectangle(img, top_left, bottom_right, (1, 1, 1), -1)


def crop_borders(img):
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    gray = 255 * (gray < 128).astype(np.uint8)  # To invert the text to white
    gray = cv2.morphologyEx(gray, cv2.MORPH_OPEN, np.ones((2, 2), dtype=np.uint8))  # Perform noise filtering
    canny = cv2.Canny(gray, 0, 150)
    coords = cv2.findNonZero(canny)  # Find all non-zero points (text)
    x, y, w, h = cv2.boundingRect(coords)  # Find minimum spanning bounding box
    rect = img[y:y + h, x:x + w + 20]  # Crop the image - note we do this on the original image
    return rect


img = cv2.imread('images/res5.jpg')
remove_templates(img)
img = crop_borders(img)

cv2.imwrite('output/op1.png', img)
cv2.imwrite('output/op2.png', cv2.cvtColor(img, cv2.COLOR_BGR2GRAY))


height = img.shape[0]
width = img.shape[1]
# Cut the image in half
width_cutoff = (width // 2)
left = img[:, :width_cutoff+5]
right = img[:, width_cutoff+25:]


cv2.imwrite('output/left.png', left)
cv2.imwrite('output/right.png', right)

The above code does give me the first result, but fails when the logo's are of different aspect ratio or sizes.

How can i achieve the same, any help will be very helpful.

I am new to opencv, so any direction will be helpful. Most of the code i have right now is picking parts from different tutorials. If there are issues in the code, plz guide me.

Answer 1

The Concept

1. Define a function to process a BGR image to a binary image with edges of the boxes enhanced.

2. Define a function that takes in a BGR image and returns the contours detected from the image (processed with the previous function) that are within a specific area range.

3. Draw the bounding boxes of each contour, and to crop the image, concatenate all the contours and get the bounding box of all the contours to use to slice the image.

The Code

import cv2
import numpy as np

def process(img):
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    img_gray[img_gray < 5] = 255
    return cv2.dilate(cv2.Canny(img_gray, 50, 75), np.ones((4, 4)), iterations=2)

def get_cnts(img):
    cnts, _ = cv2.findContours(process(img), cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    return [cnt for cnt in cnts if 80000 > cv2.contourArea(cnt) > 40000]
    
img = cv2.imread("image.png")
cnts = get_cnts(img)

for cnt in cnts:
    x, y, w, h = cv2.boundingRect(cnt)
    cv2.rectangle(img, (x, y), (x + w, y + h), (0, 255, 0), 3)

x, y, w, h = cv2.boundingRect(np.concatenate(cnts))
cv2.imshow("Image", img[y: y + h, x: x + w])

cv2.waitKey(0)
cv2.destroyAllWindows()

The Output

Here are the resulting images for both of the sample images provided:

The Explanation

1. Import all the necessary modules:

import cv2
import numpy as np

2. Define a function, process(), that takes in a BGR image array as its argument:

def process(img):

3. The processing begins with converting the image to grayscale. Then we replace every value in the grayscale array that is smaller than 5 to a greater number (I used 255). The reason for this is to lighten the background of the image so that the edges of the boxes will be easier to detect:

    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    img_gray[img_gray < 5] = 255

4. We can now use the canny edge detector to detect the edges of the boxes. 2 iterations of dilation will nicely enhance the detected edges. Then return the dilated image (which is in binary format):

    return cv2.dilate(cv2.Canny(img_gray, 50, 75), np.ones((4, 4)), iterations=2)

5. Define a function, get_pts(), that takes in a BGR image array as its argument:

def get_cnts(img):

6. Using the cv2.findContours() method, we find the contours of the image (processed using the process() function we defined earlier), and return a list of all the contours that are above 40000 in area and below 80000 in area. Significantly different box sizes will require to adjust these values:

    cnts, _ = cv2.findContours(process(img), cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    return [cnt for cnt in cnts if 80000 > cv2.contourArea(cnt) > 40000]

7. Read in the image file, get its contours, and draw the bounding rectangles of each contour using the cv2.boundingRect() and cv2.rectangle() methods:

img = cv2.imread("image.png")
cnts = get_cnts(img)

for cnt in cnts:
    x, y, w, h = cv2.boundingRect(cnt)
    cv2.rectangle(img, (x, y), (x + w, y + h), (0, 255, 0), 3)

8. Finally, to crop the image, get the bounding rectangle of all the contours in the list combined (which can be done by concatenating the contours with the np.concatenate() method) and show the result:

x, y, w, h = cv2.boundingRect(np.concatenate(cnts))
cv2.imshow("Image", img[y: y + h, x: x + w])

cv2.waitKey(0)
cv2.destroyAllWindows()