Quickly compute average color of all the clusters generated by connectedComponents

Question

I split an image into clusters with cv2.connectedComponentWithStats. Now I want to observe color changes in each of the clusters.

Currently my code looks like this:

#...Calculation of the mask

res,labels,stats,centroids = cv2.connectedComponentsWithStats(im_mask)

def compute_average(frame,i):
   data=frame[labels==i].mean(axis=0)
   return (data[2]-data[1]) # difference between red and green channel is meaningful for me

while True:
   frame=capture.read()
   if(not frame[0]):
      break
   start_time=time.time()
   measurements = [ compute_average(frame,i) for i in range(1,len(centroids))]
   print("Computing took",time.time()-start_time

It appears that calculating measurements took almost 1.5 second for each frame (I have approximately 300 clusters of 200-600 pixels each). This is unacceptable.

It seems that by cleverly choosing numpy algorithm for computing average I can get much better performance. In particular it should be possible to compute average for all clusters simultaneously. However, I'm stuck here.

Is there a way to sort pixels of image into groups according to their label?

Answer 1

Seems like a perfect use-case to leverage np.bincount which is a pretty efficient way to compute binned summations and weighted summations with its optional second argument. In our case here, we will use the labels as bins and get the summations as the counts and then frame as the weights as the optional weights arg.

Hence, we will have a vectorized and hopefully more efficient way, like so -

def bincount_method(frame, labels):
    f = frame.reshape(-1,3)
    cn1 = np.bincount(labels.ravel(), f[:,1])
    cn2 = np.bincount(labels.ravel(), f[:,2])
    return (cn2[1:]-cn1[1:])/stats[1:,-1]

Benchmarking

For the timings, we will re-use @Dan Mašek's test-setup image. We are using Python 3.6.8.

import numpy as np
import cv2
import urllib.request as ur

# Setup
url = 'https://i.sstatic.net/puxMo.png'
s = ur.urlopen(url)
url_response = ur.urlopen(url)
img_array = np.array(bytearray(url_response.read()), dtype=np.uint8)
img = cv2.imdecode(img_array, -1)
im_mask = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

res,labels,stats,centroids = cv2.connectedComponentsWithStats(im_mask)

np.random.seed(0)
frame = np.random.rand(im_mask.shape[0],im_mask.shape[1],3)

Timings -

# Original soln by OP
In [5]: %timeit [compute_average(frame,i) for i in range(1,len(centroids))]
2.38 s ± 116 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

# @Dan Mašek's soln
In [3]: %timeit rg_mean_diff_per_label(frame, labels)
92.5 ms ± 1.27 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

# Solution with bincount
In [4]: %timeit bincount_method(frame, labels)
30.1 ms ± 82.7 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

---

Allowing pre-computing on labels

Well the bincount one has a small modification for that :

L = (labels.ravel()[:,None]+np.arange(2)*res).ravel()

def bincount_method_with_precompute(frame, L):
    p = np.bincount(L,frame[...,1:].ravel())
    return (p[res+1:]-p[1:res])/stats[1:,-1]

Comparing against @Dan Mašek's solution with pre-compute on the same setup :

In [4]: %timeit bincount_method_with_precompute(frame, L)
25.1 ms ± 326 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [5]: %timeit rg_mean_diff_per_label(frame, label_indices)
20.3 ms ± 432 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)