Get Frame when detecting text using a pipeline in Android Mobile Vision

Question

I am detecting text using a pipeline just like the Google CodeLabs sample code. How do I get the preview Frame that the CameraSource sends to the TextRecognizer?

Answer 1

The preview frame is not sent beyond the text recognizer. However, you could make a class that wraps the text recognizer, receiving the preview frame before detection. See a similar discussion here.

First, implement a detector class to wrap the text recognizer:

class MyTextRecognizer extends Detector<TextBlock> {
  private Detector<TextBlock> mDelegate;

  MyTextRecognizer(Detector<TextBlock> delegate) {
    mDelegate = delegate;
  }

  public SparseArray<TextBlock> detect(Frame frame) {
    // *** add your code to access the preview frame here
    return mDelegate.detect(frame);
  }

  public boolean isOperational() {
    return mDelegate.isOperational();
  }

  public boolean setFocus(int id) {
    return mDelegate.setFocus(id);
  }
}

Wrap the text recognizer with your class, and pass your class into the camera source. It would look something like this:

TextRecognizer textRecognizer = new TextRecognizer.Builder(context)
        .build();
TextRecognizer myTextRecognizer = new MyTextRecognizer(textRecognizer);

myTextRecognizer.setProcessor(/* include your processor here */);

mCameraSource = new CameraSource.Builder(context, myTextRecognizer)
        .build();

Your MyTextRecognizer will be called first with the raw frame data.

Note that the image may not be upright, if the device is rotated. You can get the orientation through the frame's metadata.getRotation method.

One word of caution: once the detect method returns, you should not access the frame pixel data. Since the camera source recycles image buffers, the contents of the frame object will be eventually overridden once the method returns.