Disparity map from foreground masked images

Question

I am trying to extract the disparity map of foreground objects in an image. The foreground objects are extracted using color and the final purpose is to determine the coordinates of the extracted objects. Below is the masked left image of the view with reddish objects extracted

and then there is the right image

The background is basically a giant window that I want to be ignored and I only care to find the position of the reddish (or any color I later choose) objects.

After playing around with the parameters of the SGBM algorithm in the OpenCV example, mainly

int SADWindowSize
int minDisparity
int numberOfDisparities

I was not able to get satisfying results, more exactly the algorithm was not able to deal very well with the uniform texture of the masked parts. I will post two examples to illustrate. The SADWindowSize is the only parameter varied in those examples because it gives the most distinctive results.

Example 1: with smaller window size = 9 and number of disparities = 64

Example 2: with smaller window size = 23 and number of disparities = 64

The bigger window size gives more smeared results that are undesirable.

The question: Is it a wrong approach to mask the background when calculating the disparity map? Another possible approach is to calculate the disparity map then apply the mask but I am not sure about the plausibility of the results in this case.

Note that the cameras and calibrated and the images (and masks) are rectified.

Answer 1

Masking before calculating depth map doesn't have sense, because algorithm need to compare specified space to find corresponding pixels. Using mask cause lack of information due to lot of black pixels. So, what you trying is intuitive for us, but application can't easily determine which pixels represent the same point.

I'm not sure but if you use mask for left view on disparity you should get what you're expecting. Or mask an output from reprojectImageTo3D().