How can I cut a green background with the foreground from the rest of the picture in Python?

Question

I'm trying to cut multiple images with a green background. The center of the pictures is green and i want to cut the rest out of the picture. The problem is, that I got the pictures from a video, so sometimes the the green center is bigger and sometimes smaller. My true task is to use K-Means on the knots, therefore i have for example a green background and two ropes, one blue and one red.

I use python with opencv, numpy and matplotlib.

I already cut the center, but sometimes i cut too much and sometimes i cut too less. My Imagesize is 1920 x 1080 in this example.

Here the knot is left and there is more to cut

Here the knot is in the center

Here is another example

Here is my desired output from picture 1

Example 1 which doesn't work with all algorithm

Example 2 which doesn't work with all algorithm

Example 3 which doesn't work with all algorithm

Here is my Code so far:

import numpy as np
import cv2
import matplotlib.pyplot as plt
from PIL import Image, ImageEnhance

img = cv2.imread('path')

print(img.shape)

imgRGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

crop_img = imgRGB[500:500+700, 300:300+500]

plt.imshow(crop_img)
plt.show()

Answer 1

The code below does what you want. First it converts the image to the HSV colorspace, which makes selecting colors easier. Next a mask is made where only the green parts are selected. Some noise is removed and the rows and columns are summed up. Finally a new image is created based on the first/last rows/cols that fall in the green selection.

Since in all provided examples a little extra of the top needed to be cropped off I've added code to do that. First I've inverted the mask. Now you can use the sum of the rows/cols to find the row/col that is fully within the green selection. It is done for the top. In the image below the window 'Roi2' is the final image.

Edit: updated code after comment by ts.
Updated result:

Code:

import numpy as np 
import cv2

# load image
img = cv2.imread("gr.png")
# convert to HSV
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV) 
# set lower and upper color limits
lower_val = (30, 0, 0)
upper_val = (65,255,255)
# Threshold the HSV image to get only green colors
# the mask has white where the original image has green
mask = cv2.inRange(hsv, lower_val, upper_val)
# remove noise
kernel =  np.ones((8,8),np.uint8)
mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)

# sum each row and each volumn of the image
sumOfCols = np.sum(mask, axis=0)
sumOfRows = np.sum(mask, axis=1)

# Find the first and last row / column that has a sum value greater than zero, 
# which means its not all black. Store the found values in variables
for i in range(len(sumOfCols)):
    if sumOfCols[i] > 0:
        x1 = i
        print('First col: ' + str(i))
        break

for i in range(len(sumOfCols)-1,-1,-1):
    if sumOfCols[i] > 0:
        x2 = i
        print('Last col: ' + str(i))
        break

for i in range(len(sumOfRows)):
    if sumOfRows[i] > 0:
        y1 = i
        print('First row: ' + str(i))
        break

for i in range(len(sumOfRows)-1,-1,-1):
    if sumOfRows[i] > 0:
        y2 = i
        print('Last row: ' + str(i))
        break

# create a new image based on the found values
#roi = img[y1:y2,x1:x2]

#show images
#cv2.imshow("Roi", roi)



# optional: to cut off the extra part at the top:
#invert mask, all area's not green become white
mask_inv = cv2.bitwise_not(mask)
# search the first and last column top down for a green pixel and cut off at lowest common point
for i in range(mask_inv.shape[0]):
    if mask_inv[i,0] == 0 and mask_inv[i,x2] == 0:
        y1 = i
        print('First row: ' + str(i))
        break

# create a new image based on the found values
roi2 = img[y1:y2,x1:x2]

cv2.imshow("Roi2", roi2)
cv2.imwrite("img_cropped.jpg", roi2)
cv2.waitKey(0)
cv2.destroyAllWindows()