Obtain complete pattern of shapes with OpenCV

Question

I'm working in a script using different OpenCV operations for processing an image with solar panels in a house roof. My original image is the following:

After processing the image, I get the edges of the panels as follows:

It can be seen how some rectangles are broken due to reflection of the Sun in the picture.

I would like to know if it's possible to fix those broken rectangles, maybe by using the pattern of those which are not broken.

My code is the following:

# Load image
color_image = cv2.imread("google6.jpg")
cv2.imshow("Original", color_image)
# Convert to gray
img = cv2.cvtColor(color_image, cv2.COLOR_BGR2GRAY)

# Apply various filters
img = cv2.GaussianBlur(img, (5, 5), 0)
img = cv2.medianBlur(img, 5)
img = img & 0x88 # 0x88
img = cv2.fastNlMeansDenoising(img, h=10)

# Invert to binary
ret, thresh = cv2.threshold(img, 127, 255, 1) 

# Perform morphological erosion
kernel = np.ones((5, 5),np.uint8)
erosion = cv2.morphologyEx(thresh, cv2.MORPH_ERODE, kernel, iterations=2)

# Invert image and blur it
ret, thresh1 = cv2.threshold(erosion, 127, 255, 1)
blur = cv2.blur(thresh1, (10, 10))

# Perform another threshold on blurred image to get the central portion of the edge
ret, thresh2 = cv2.threshold(blur, 145, 255, 0)

# Perform morphological erosion to thin the edge by ellipse structuring element
kernel1 = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(5,5))
contour = cv2.morphologyEx(thresh2, cv2.MORPH_ERODE, kernel1, iterations=2)

# Get edges
final = cv2.Canny(contour, 249, 250)
cv2.imshow("final", final)

I have tried to modify all the filters I'm using in order to reduce as much as possible the effect of the Sun in the original picture, but that is as far as I have been able to go.

I'm in general happy with the result of all those filters (although any advice is welcome), so I'd like to work on the black/white imaged I showed, which is already smooth enough for the post-processing I need to do.

Thansk!

Answer 1

The pattern is not broken in the original image, so it being broken in your binarized result must mean your binarization is not optimal.

You apply threshold() to binarize the image, and then Canny() to the binary image. The problems here are:

1. Thresholding removes a lot of information, this should always be the last step of any processing pipeline. Anything you lose here, you've lost for good.
2. Canny() should be applied to a gray-scale image, not a binary image.
3. The Canny edge detector is an edge detector, but you want to detect lines, not edges. See here for the difference.

So, I suggest starting from scratch.

The Laplacian of Gaussian is a very simple line detector. I took these steps:

1. Read in image, convert to grayscale.
2. Apply Laplacian of Gaussian with sigma = 2.
3. Invert (negate) the result and then set negative values to 0.

This is the output:

From here, it should be relatively straight-forward to identify the grid pattern.

I don't post code because I used MATLAB for this, but you can accomplish the same result in Python with OpenCV, here is a demo for applying the Laplacian of Gaussian in OpenCV.

---

This is Python + OpenCV code to replicate the above:

import cv2

color_image = cv2.imread("/Users/cris/Downloads/L3RVh.jpg")
img = cv2.cvtColor(color_image, cv2.COLOR_BGR2GRAY)
out = cv2.GaussianBlur(img, (0, 0), 2)  # Note! Specify size of Gaussian by the sigma, not the kernel size
out = cv2.Laplacian(out, cv2.CV_32F)
_, out = cv2.threshold(-out, 0, 1e9, cv2.THRESH_TOZERO)

However, it looks like OpenCV doesn't linearize (apply gamma correction) when converting from BGR to gray, as the conversion function does that I used when creating the image above. I think this gamma correction might have improved the results a bit by reducing the response to the roof tiles.