How to separate human body from background in an image

Question

I have been trying to separate the human body in an image from the background, but all the methods I have seen don't seem to work very well for me.

I have collected the following images;

• The image of the background
• The image of the background with the person in it.

Now I want to cut out the person from the background. I tried subtracting the image of the background from the image with the person using res = cv2.subtract(background, foreground) (I am new to image processing).

Background subtraction methods in opencv like cv2.BackgroundSubtractorMOG2() and cv2.BackgroundSubtractorMOG2() only works with videos or image sequence and contour detection methods I have seen are only for solid shapes.

And grabCut doesn't quite work well for me because I would like to automate the process.

Given the images I have (Image of the background and image of the background with the person in it), is there a method of cutting the person out from the background?

Answer 1

I wouldn't recommend a neural net for this problem. That's a lot of work for something like this where you have a known background. I'll walk through the steps I took to do the background segmentation on this image.

First I shifted into the LAB color space to get some light-resistant channels to work with. I did a simple subtractions of foreground and background and combined the a and b channels.

You can see that there is still significant color change in the background even with a less light-sensitive color channel. This is likely due to the auto white balance on the camera, you can see that some of the background colors change when you step into view.

The next step I took was thresholding off of this image. The optimal threshold values may not always be the same, you'll have to adjust to a range that works well for your set of photos.

I used openCV's findContours function to get the segmentation points of each blob and I filtered the available contours by size. I set a size threshold of 15000. For reference, the person in the image had a pixel area of 27551.

Then it's just a matter of cropping out the contour.

This technique works for any good thresholding strategy. If you can improve the consistency of your pictures by turning off auto settings and ensure good contrast of the person against the wall then you can use simpler thresholding strategies and get good results.

Just for fun:

Edit:

I forgot to add in the code I used:

import cv2
import numpy as np

# rescale values
def rescale(img, orig, new):
    img = np.divide(img, orig);
    img = np.multiply(img, new);
    img = img.astype(np.uint8);
    return img;

# get abs(diff) of all hue values
def diff(bg, fg):
    # do both sides
    lh = bg - fg;
    rh = fg - bg;

    # pick minimum # this works because of uint wrapping
    low = np.minimum(lh, rh);
    return low;

# load image
bg = cv2.imread("back.jpg");
fg = cv2.imread("person.jpg");
fg_original = fg.copy();

# blur
bg = cv2.blur(bg,(5,5));
fg = cv2.blur(fg,(5,5));

# convert to lab
bg_lab = cv2.cvtColor(bg, cv2.COLOR_BGR2LAB);
fg_lab = cv2.cvtColor(fg, cv2.COLOR_BGR2LAB);
bl, ba, bb = cv2.split(bg_lab);
fl, fa, fb = cv2.split(fg_lab);

# subtract
d_b = diff(bb, fb);
d_a = diff(ba, fa);

# rescale for contrast
d_b = rescale(d_b, np.max(d_b), 255);
d_a = rescale(d_a, np.max(d_a), 255);

# combine
combined = np.maximum(d_b, d_a);

# threshold 
# check your threshold range, this will work for
# this image, but may not work for others
# in general: having a strong contrast with the wall makes this easier
thresh = cv2.inRange(combined, 70, 255);

# opening and closing
kernel = np.ones((3,3), np.uint8);

# closing
thresh = cv2.dilate(thresh, kernel, iterations = 2);
thresh = cv2.erode(thresh, kernel, iterations = 2);

# opening
thresh = cv2.erode(thresh, kernel, iterations = 2);
thresh = cv2.dilate(thresh, kernel, iterations = 3);

# contours
_, contours, _ = cv2.findContours(thresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE);

# filter contours by size
big_cntrs = [];
marked = fg_original.copy();
for contour in contours:
    area = cv2.contourArea(contour);
    if area > 15000:
        print(area);
        big_cntrs.append(contour);
cv2.drawContours(marked, big_cntrs, -1, (0, 255, 0), 3);

# create a mask of the contoured image
mask = np.zeros_like(fb);
mask = cv2.drawContours(mask, big_cntrs, -1, 255, -1);

# erode mask slightly (boundary pixels on wall get color shifted)
mask = cv2.erode(mask, kernel, iterations = 1);

# crop out
out = np.zeros_like(fg_original) # Extract out the object and place into output image
out[mask == 255] = fg_original[mask == 255];

# show
cv2.imshow("combined", combined);
cv2.imshow("thresh", thresh);
cv2.imshow("marked", marked);
# cv2.imshow("masked", mask);
cv2.imshow("out", out);
cv2.waitKey(0);