ImageReader format compatibility for FirebaseVisionImageMetadata (via Camera2 API on Android)

Question

I'd like to use the image byte array from the ImageReader to construct a FirebaseVisionImageMetadata object for barcode scanning (as part of the ML Kit ). I'm using the Camera2 API and following this PR for supporting frame previews.

Size sz = chooseOptimalSize();
mCallbackImageDataReader =
        ImageReader.newInstance(sz.getWidth(), sz.getHeight(), ImageFormat.YV12, 1);
    mCallbackImageDataReader.setOnImageAvailableListener(reader -> {
        byte[] data = getByteDataFromImageReader(reader);
        if (data != null) {
             //rotation compensation calculation not shown
            mCallback.onPreviewFrame(data, reader.getImageFormat(), reader.getWidth(), reader.getHeight(), rotationCompensation);
        }
    }, null);

@Nullable
private byte[] getByteDataFromImageReader(@NonNull ImageReader reader) {
    byte[] data = null;
    try (Image image = reader.acquireNextImage()) {
        Image.Plane[] planes = image.getPlanes();
        if (planes.length > 0) {
            ByteBuffer buffer = planes[0].getBuffer();
            data = new byte[buffer.remaining()];
            buffer.get(data);
        }
    }
    return data;
} 

However, when I pass in the resulting format to the FirebaseVisionImageMetadata builder, I get the following error:

final FirebaseVisionImageMetadata meta = new FirebaseVisionImageMetadata.Builder()
        .setFormat(format) //exception thrown here
        .setWidth(width)
        .setHeight(height)
        .setRotation(rotation)
        .build();

java.lang.IllegalArgumentException
    at com.google.android.gms.common.internal.Preconditions.checkArgument(Unknown Source:5)
    at com.google.firebase.ml.vision.common.FirebaseVisionImageMetadata$Builder.setFormat(Unknown Source:13)

The format I’m passing in to the ImageReader is ImageReader.YV12 since FirebaseVisionImageMetadata only accepts either NV21 or YV12, but since ImageReader doesn’t accept NV21, I’m left with YV12. When I use this format (as shown above), the camera launches with a black screen. I've also tried passing in ImageReader.YUV_420_888, but that results in the same IllegalArgumentException from FirebaseVisionImageMetadata.

My guess is that I probably have to convert the byte array differently, or to a different format (?).

Any help would be appreciated.

Sources:

1. https://github.com/google/cameraview/pull/123/commits/f90580b1891c3edf91b4a254219d08b01110afce
2. https://developer.android.com/reference/android/graphics/ImageFormat.html?hl=BG#YUV_420_888

Answer 1

I figured it out. I needed to use reader.acquireNextImage();

mCallbackImageDataReader.setOnImageAvailableListener(reader -> {
        try (Image image = reader.acquireNextImage())
        {
            if (image != null)
            {
                //compute rotationCompensation
                mCallback.onPreviewFrame(image, rotationCompensation);
            }
        }

    }, null);