Search code examples
google-cloud-vision

How to detect a cropped-off head with Google Vision API?

How can the Google Vision API be used to detect if a head is completely inside an image or partly cut off by the image frame?

3 examples:

  • Image 1 shows a complete head
  • Image 2 shows a cut off head where the full "face" is visible
  • Image 3 shows a cut off head where also the face is cut off

enter image description here

To narrow down the question, the following cases should be detected:

  • there is a completely visible head in the image
  • there is a partly visible head in the image where parts of the head are outside the image bounds

The following is out of scope for this question:

  • heads that are spatially or scenically inside the image bounds but fully or partly covered by other objects
  • there are no parts of a head visible in the image, e.g. if there is only a neck visible it can't be assumed that there is or is not a head attached to it
  • the effectiveness or efficiency of the API in detecting faces that are fully or partly visible, file that under caveats

I have checked the documentation but it doesn't say anything about head crop-off detection.

I am not asking for code but whether / how the API can be used for the described purpose. Hence neither the question contains any code nor is an answer expected to contain any code. If you are looking for code examples for API calls, take a look at the plenty example calls in the API docs.

There was a meta discussion about this question.

Image credit

Solution

  • The API does provide boundingPoly and fdBoundingPoly to determine where the face (not head) is located inside the image.

    Even though the API does not provide the bounds for the "head", the example below demonstrates that boundingPoly is pretty equivalent to the larger face area which can apparently be used as substitute.

    Here are the API results for boundingPoly (red dots) and fdBoundingPoly (blue dots):

    enter image description here

    • Image 1: shows the face bounds for a fully visible head. The boundingPoly which is described as the larger face area is apparently pretty equivalent to the "head" area.
    • Image 2: the bounding boxes are missing y coordinates for the 4 top dots (2 red, 2 blue; depicted with y=0 coordinates for the purpose of demonstration), which can be used as indication of a cropped-off head.
    • Image 3: the API does not result any faceAnnotations, so it can be easily detected as a substantially cropped-off head or an image without any partly visible head which won’t be distinguished in this case.