How to remove the background from an image

Question

I want to remove the background, and draw the outline of the box shown in the image(there are multiple such images with a similar background) . I tried multiple methods in OpenCV, however I am unable to determine the combination of features which can help remove background for this image. Some of the approaches tried out were:

• Edge Detection - Since the background itself has edges of its own, using edge detection on its own (such as Canny and Sobel) didn't seem to give good results.
• Channel Filtering / Thresholding - Both the background and foreground have a similar white color, so I was unable to find a correct threshold to filter the foreground.
• Contour Detection - Since the background itself has a lot of contours, just using the largest contour area, as is often used for background removal, also didn't work.

I would be open to tools in Computer Vision or of Deep Learning (in Python) to solve this particular problem.

Answer 1

The Concept

This is one of the cases where it is really useful to fine-tune the kernels of which you are using to dilate and erode the canny edges detected from the images. Here is an example, where the dilation kernel is np.ones((4, 2)) and the erosion kernel is np.ones((13, 7)):

The Code

import cv2
import numpy as np

def process(img):
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    img_blur = cv2.GaussianBlur(img_gray, (3, 3), 2)
    img_canny = cv2.Canny(img_blur, 50, 9)
    img_dilate = cv2.dilate(img_canny, np.ones((4, 2)), iterations=11)
    img_erode = cv2.erode(img_dilate, np.ones((13, 7)), iterations=4)
    return cv2.bitwise_not(img_erode)

def get_contours(img):
    contours, _ = cv2.findContours(process(img), cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    cnt = max(contours, key=cv2.contourArea)
    cv2.drawContours(img, [cv2.convexHull(cnt)], -1, (0, 0, 255), 2)

img = cv2.imread("image2.png")
get_contours(img)
cv2.imshow("result", img)

cv2.waitKey(0)
cv2.destroyAllWindows()

The Output

Output for each of the two images provided:

Image 1:

Image 2:

Notes

Note that the processed image (which is binary) is inverted at cv2.bitwise_not(img_erode). Observe the processed version of both images (returned by the process() function defined above), with the inversion:

Processed Image 1:

Processed Image 2:

Tools

Finally, if you happen to have other images where the above program doesn't work properly on, you can use OpenCV Trackbars to adjust the values passed into the methods with the program below:

import cv2
import numpy as np

def process(img, b_k, b_s, c_t1, c_t2, k1, k2, k3, k4, iter1, iter2):
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    b_k = b_k // 2 * 2 + 1
    img_blur = cv2.GaussianBlur(img_gray, (b_k, b_k), b_s)
    img_canny = cv2.Canny(img_blur, c_t1, c_t2)
    img_dilate = cv2.dilate(img_canny, np.ones((k1, k2)), iterations=iter1)
    img_erode = cv2.erode(img_dilate, np.ones((k3, k4)), iterations=iter2)
    return cv2.bitwise_not(img_erode)

d = {"Blur Kernel": (3, 50),
     "Blur Sigma": (2, 30),
     "Canny Threshold 1": (50, 500),
     "Canny Threshold 2": (9, 500),
     "Dilate Kernel1": (4, 50),
     "Dilate Kernel2": (2, 50),
     "Erode Kernel1": (13, 50),
     "Erode Kernel2": (7, 50),
     "Dilate Iterations": (11, 40),
     "Erode Iterations": (4, 40)}

cv2.namedWindow("Track Bars")
for i in d:
    cv2.createTrackbar(i, "Track Bars", *d[i], id)

img = cv2.imread("image1.png")

while True:
    img_copy = img.copy()
    processed = process(img, *(cv2.getTrackbarPos(i, "Track Bars") for i in d))
    contours, _ = cv2.findContours(processed, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    if contours:
        cnt = max(contours, key=cv2.contourArea)
        cv2.drawContours(img_copy, [cv2.convexHull(cnt)], -1, (0, 0, 255), 2)
    cv2.imshow("result", img_copy)
    
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cv2.waitKey(0)
cv2.destroyAllWindows()