get bounding boxes with maximum confidence pandas opencv python
I have a Symbol detection algorithm, which can be output from template matching/ faster rcnn or combining the results from both of them, which gives me the coordinates filename,xmin, ymin, xmax, ymax, class, confidence.
The issue is that there are multiple bounding boxes occurring for the same object. How I can efficiently filter these boxes and get only the boxes with maximum confidence in the area removing duplicates.
sample Image:
The sample coordinate data frame:
df=pd.DataFrame({'filename':['dummyfile.jpg']*5, class=['cube']*5, xmin':[88,87,65,492,470],'ymin':[87,111,110,187,184],'xmax':[197,198,174,603,578],
'ymax':[198,220,221,295,295],'confidence':[0.99,0.88,0.95,0.89,0.83]})
class confidence filename xmax xmin ymax ymin
0 cube 0.99 dummyfile.jpg 197 88 198 87
1 cube 0.88 dummyfile.jpg 198 87 220 111
2 cube 0.95 dummyfile.jpg 174 65 221 110
3 cube 0.89 dummyfile.jpg 603 492 295 187
4 cube 0.83 dummyfile.jpg 578 470 295 184
Image representation:
Expected output:
I tried filtering using confidence as a threshold, but it will affect the recall of the solution. How to remove these duplicates making use of IoU?
This is the solution which I came up with.
Creating a unique key for each bounding box
df['key']=df['xmin'].astype(str)+'_'+df['ymin'].astype(str)+'_'+df['xmax'].astype(str)+'_'+df['ymax'].astype(str)
Making an outer join of all the rows based on filename. (for calculating IoU)
###copy df
df_1=df.copy()
###Renaming df columns with _1 suffix
df_cols=df.columns.tolist()
df_cols.remove('filename')
new_cols=[col+'_1' for col in df_cols]
new_col_dict=dict(zip(df_cols,new_cols))
df_1.rename(columns=new_col_dict,inplace=True)
### Outer joining both dataframes
newdf=pd.merge(df,df_1,'outer',on='filename')
Outer join sample:
Function to find IoU of each row
def IOU(df):
'''funtion to calulcate IOU within rows of dataframe'''
# determining the minimum and maximum -coordinates of the intersection rectangle
xmin_inter = max(df.xmin, df.xmin_1)
ymin_inter = max(df.ymin, df.ymin_1)
xmax_inter = min(df.xmax, df.xmax_1)
ymax_inter = min(df.ymax, df.ymax_1)
# calculate area of intersection rectangle
inter_area = max(0, xmax_inter - xmin_inter + 1) * max(0, ymax_inter - ymin_inter + 1)
# calculate area of actual and predicted boxes
actual_area = (df.xmax - df.xmin + 1) * (df.ymax - df.ymin + 1)
pred_area = (df.xmax_1 - df.xmin_1 + 1) * (df.ymax_1 - df.ymin_1+ 1)
# computing intersection over union
iou = inter_area / float(actual_area + pred_area - inter_area)
# return the intersection over union value
return iou
Calculating IoU for each row and filtering rows with ioU<0.4
newdf['iou']= newdf.apply(IOU, axis = 1)
### filtering all iou<0.4
newdf=newdf[newdf['iou']>=0.4]
Getting the best match for each key
once we have the IoU match dataframe, parse through each unique key bounding box, get the value with max confidence for each unique key with iou>0.4
best_df=pd.DataFrame()
for i, v in df.iterrows():
key=v['key']
iou_match=newdf[newdf['key']==key]
iou_match.sort_values('confidence_1',ascending=False,inplace=True)
iou_match=iou_match.reset_index()
best_match=iou_match.loc[0,['filename','class_1','xmin_1','ymin_1','xmax_1','ymax_1','confidence_1']]
best_df=best_df.append(best_match,ignore_index=True)
best_df now looks like:
class_1 confidence_1 filename xmax_1 xmin_1 ymax_1 ymin_1
0 cube 0.99 dummyfile.jpg 197.0 88.0 198.0 87.0
1 cube 0.99 dummyfile.jpg 197.0 88.0 198.0 87.0
2 cube 0.99 dummyfile.jpg 197.0 88.0 198.0 87.0
3 cube 0.89 dummyfile.jpg 603.0 492.0 295.0 187.0
4 cube 0.89 dummyfile.jpg 603.0 492.0 295.0 187.0
Removing duplicates to get unique best matches
best_df.drop_duplicates(inplace=True)
Final result:
class_1 confidence_1 filename xmax_1 xmin_1 ymax_1 ymin_1
0 cube 0.99 dummyfile.jpg 197.0 88.0 198.0 87.0
3 cube 0.89 dummyfile.jpg 603.0 492.0 295.0 187.0