Search code examples
tensorflowobject-detectionevaluationobject-detection-api

Is a mislabeled box a False Positive in Pascal VOC metrics?

I am using Tensorflow object-detection API for training a custom object detector. I have decided to use Pascal VOC 2010-2012 as my evaluation metric. Everything is working as it is supposed to, but I have an issue understanding the idea behind false positive results. According to the official document for Pascal VOC challenge, a false positive is when the IoU is smaller than the threshold(.5 for example). However, I couldn't find any source which mentions whether a mislabeled box with a high IoU is also considered a false positive or it will be classified as an ignored box (False Negative). Anyone knows where can I find some more data on this matter?

Solution

  • According to the definition found at: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/evaluation_protocols.md

    A detection is a "false positive" if it is neither a "true positive" nor "ignored".

    Have a look at this example(also borrowed from the site above):

    In the picture below, it is clear that,for the upper left detection(where the detection is a car but the ground truth is a bus, the IoU metric is higher that 0.5).

    However, due to the fact that the ground truth is a bus, yet the detection is a car, it is also considered a false positive.

    ***NOTE: Although the example above belongs to OID examples and not VOC, since the mAP is used in VOC evaluation, the same criteria should apply.

    enter image description here

    According to the snippet from below(which is taken from : http://host.robots.ox.ac.uk/pascal/VOC/voc2010/devkit_doc_08-May-2010.pdf)

    Detections are considered true or false positives based on the area of overlap with ground truth bounding boxes. To be considered a correct detection, the area of overlap ao between the predicted bounding box Bp and ground truth bounding box Bgt must exceed 50%.

    As the notion of False Negative is non-existent in case of object-detection(only at the segmentation level if you read the evaluation chapters for both object detection and segmentation), it is clear that the case that you outlined is considered a false positive.