Reduce tracking window using google mlkit vision samples

Question

I would like to reduce the reduce bar code tracking window when using the google vision api. There are some answers here but they feel a bit outdated.

I'm using google's sample: https://github.com/googlesamples/mlkit/tree/master/android/vision-quickstart

Currently, I try to figure out if a barcode is inside my overlay box inside BarcodeScannerProcessor onSuccess callback:

override fun onSuccess(barcodes: List<Barcode>, graphicOverlay: GraphicOverlay) {
    if(barcodes.isEmpty())
      return;

    for(barcode in barcodes) {
      val center = Point(graphicOverlay.imageWidth / 2, graphicOverlay.imageHeight / 2)
      val rectWidth = graphicOverlay.imageWidth * Settings.OverlayWidthFactor
      val rectHeight = graphicOverlay.imageHeight * Settings.OverlayHeightFactor

      val left = center.x - rectWidth / 2
      val top = center.y - rectHeight / 2
      val right = center.x + rectWidth / 2
      val bottom = center.y + rectHeight / 2

      val rect = Rect(left.toInt(), top.toInt(), right.toInt(), bottom.toInt())

      val contains = rect.contains(barcode.boundingBox!!)
      val color = if(contains) Color.GREEN else Color.RED

      graphicOverlay.add(BarcodeGraphic(graphicOverlay, barcode, "left: ${barcode.boundingBox!!.left}", color))
    }
}

Y-wise it works perfectly, but the X values from barcode.boundingBox e.g. barcode.boundingBox.left seems to have an offset. Is it based on what's being calculated in GraphicOverlay?

I'm expecting the value below to be close to 0, but the offset is about 90 here:

Or perhaps it's more efficient to crop the image according to the box?

Answer 1

Actually the bounding box is correct. The trick is that the image aspect ratio doesn't match the viewport aspect ratio so the image is cropped horizontally. Try to open settings (a gear in the top right corner) and choose an appropriate resolution.

For example take a look at these two screenshots. On the first one the selected resolution (1080x1920) matches my phone resolution so the padding looks good (17px). On the second screenshot the aspect ratio is different (1.0 for 720x720 resolution) therefore the image is cropped and the padding looks incorrect.

So the offset should be transformed from image coordinates to the screen coordinates. Under the hood GraphicOverlay uses a matrix for this transformation. You can use the same matrix:

    for(barcode in barcodes) {
      barcode.boundingBox?.let { bbox ->
        val offset = floatArrayOf(bbox.left.toFloat(), bbox.top.toFloat())
        graphicOverlay.transformationMatrix.mapPoints(offset)

        val leftOffset = offset[0]
        val topOffset = offset[1]

        ...
      }
    }

The only thing is that the transformationMatrix is private, so you should add a getter to access it.