Search code examples
androidcamera

Accessing HardwareBuffer from ImageReader

I am writing an Application for an Android device where I want to process the image from the camera.

The camera on this device only supports the NV21 and PRIVATE formats. I verified via the CameraCharacteristics and tried different formats like YUV_420_888 only for the app to fail. The camera HW Support is: INFO_SUPPORTED_HARDWARE_LEVEL_LIMITED

To create the CameraCaptureSession, I need to create an ImageReader. To create the ImageReader I need to select the ImageFormat, in this case only PRIVATE works. If I try NV21, I get an error saying that it's not supported.

My onImageAvailableListener gets triggered but since the ImageFormat is PRIVATE, the "planes" attribute in the Image returns NULL.

According to the documentation, it's possible to access the data via the HardwareBuffer.

Starting in Android P private images may also be accessed through their hardware buffers (when available) through the Image#getHardwareBuffer() method. Attempting to access the planes of a private image, will return an empty array.

I do get an object of type HardwareBuffer when I get the image from the acquireLatestImage, but my question is: How do I get the actual data/bytes that represents the pixels from the HardwareBuffer object?

Solution

  • As mentioned in HardwareBuffer documentation:

    For more information, see the NDK documentation for AHardwareBuffer

    You have to use AHardwareBuffer C language functions (using NDK) to access pixels of HardwareBuffer.

    In short:

    1. Create native JNI method in some helper class to call it from Java/Kotlin
    2. Pass your HardwareBuffer to this method as parameter
    3. Inside this method call AHardwareBuffer_fromHardwareBuffer to wrap HardwareBuffer as AHardwareBuffer
    4. Call AHardwareBuffer_lock or AHardwareBuffer_lockPlanes to obtain raw image bytes
    5. Work with pixels. If needed you can call any Java/Kotlin method to process pixels there (use ByteBuffer to wrap raw data).
    6. Call AHardwareBuffer_unlock to remove lock from buffer

    Don't forget that HardwareBuffer may be read only or protected.