How to get Bitmap from session.update() in ARCore [Android Studio]

Question

I am trying to get a Bitmap from the current frame of my ARSession with ARCore. But it always equals null. I've already been searching the web for quite a while but cannot figure out what I am doing wrong.

try {
    capturedImage = mFrame.acquireCameraImage();

    ByteBuffer buffer = capturedImage.getPlanes()[0].getBuffer();

    byte[] bytes = new byte[buffer.capacity()];

    Bitmap bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.length,null);

    if (bitmap == null) 
        Log.e(TAG,"Bitmap was NOT initialized!");

} catch(Exception e){

}

I am getting mFrame from onDrawFrame of my GLSurfaceView which I use to display the camera image. Everything works just fine except that my Bitmap equals null.

I am using a Button, so that only a single Frame is being used, as follows:

scanButton = (Button) findViewById(R.id.scanButton);
scanButton.setOnClickListener(new View.OnClickListener() {
    @Override
    public void onClick(View view) {
        checkbox = false;
        if (capturedImage!=null) capturedImage.close();
            BitmapMethod();
    }
});

capturedImage, buffer and bytes all do not equal null.

Is there probably something wrong with mFrame.acquireCameraImage()?

Thanks a lot

Answer 1

Is there probably something wrong with mFrame.acquireCameraImage()?

No, mFrame.acquireCameraImage() works as intended.

But it always equals null

The Bitmap will always equal null since bitmap factory does not understand the image data that is passed to it.

The method mFrame.acquireCameraImage() responds with an object of type Image that is in the YUV format or YCbCr. These types of images have 3 planes which is explained here very nicely. The ByteArray contained in these planes may be read directly by a CPU/GPU in native code. BitmapFactory cannot read this type of data. Hence, you need to convert this YUV image into something else.

For that, you need to use YuvImage class to create an instance of YUV & then convert it into JPEG using the compressToJpeg method. Once you have the byteArray from this, you can simply do what you're doing above. Use BitmapFactory to convert it into Bitmap and add it to your ImageView.

Note : YUV has 3 planes. Create a single bytearray from all planes & then pass it to YUV constructor. Though not elaborate, it should look something similar to this :

//The camera image received is in YUV YCbCr Format. Get buffers for each of the planes and use them to create a new bytearray defined by the size of all three buffers combined
val cameraPlaneY = cameraImage.planes[0].buffer
val cameraPlaneU = cameraImage.planes[1].buffer
val cameraPlaneV = cameraImage.planes[2].buffer

//Use the buffers to create a new byteArray that 
val compositeByteArray = ByteArray(cameraPlaneY.capacity() + cameraPlaneU.capacity() + cameraPlaneV.capacity())

cameraPlaneY.get(compositeByteArray, 0, cameraPlaneY.capacity())
cameraPlaneU.get(compositeByteArray, cameraPlaneY.capacity(), cameraPlaneU.capacity())
cameraPlaneV.get(compositeByteArray, cameraPlaneY.capacity() + cameraPlaneU.capacity(), cameraPlaneV.capacity())

val baOutputStream = ByteArrayOutputStream()
val yuvImage: YuvImage = YuvImage(compositeByteArray, ImageFormat.NV21, cameraImage.width, cameraImage.height, null)
yuvImage.compressToJpeg(Rect(0, 0, cameraImage.width, cameraImage.height), 75, baOutputStream)
val byteForBitmap = baOutputStream.toByteArray()
val bitmap = BitmapFactory.decodeByteArray(byteForBitmap, 0, byteForBitmap.size)
imageView.setImageBitmap(bitmap)

That's just a rough code. It has scope for improvement perhaps. Also refer here.