Using Vulkan to sample from Android Camera2 hardware buffer - Issue with image formats

Question

I'm currently working on an app in C++ using the Android ndk, and I need to create a sampler to access the camera output image.

I have done this using the AIMAGE_FORMAT_YUV_420_888, and using the VkSamplerYcbcrConversion for accessing the image in the hardware buffer. I do the yuv -> rgb conversion in a shader, and it all looks good on my phone.

I have since discovered that this doesn't work on Samsung phones, in my case specifically the Samsung Galaxy S10/S10+.

The reason is that when I set up an image reader with the AIMAGE_FORMAT_YUV_420_888 I get a camera error using Samsung. On my OnePlus and on another phone I tried the pipeline worked entirely as expected. I created a very simple test setup to even try to open the camera with that image format in the ImageReader on Samsung S10 and got the error, but when I changed the ImageReader format to AIMAGE_FORMAT_JPEG the error went away and the camera seemed to start as expected.

AImageReader* SimpleCamera::CreateJpegReader()
{
    AImageReader* reader = nullptr;
   // media_status_t status = AImageReader_new(640, 480, AIMAGE_FORMAT_JPEG,
    //AIMAGE_FORMAT_RGBA_8888
    //media_status_t status = AImageReader_new(640, 480, AIMAGE_FORMAT_RGB_565,4, &reader);
    media_status_t status = AImageReader_newWithUsage(640, 480, 
        //AIMAGE_FORMAT_RGBA_8888,
        //AIMAGE_FORMAT_RGB_565,
        //AIMAGE_FORMAT_RGB_888,
        AIMAGE_FORMAT_JPEG,
        AHARDWAREBUFFER_USAGE_GPU_SAMPLED_IMAGE | AHARDWAREBUFFER_USAGE_CPU_READ_RARELY,
        4, &reader);

    if (status != AMEDIA_OK) {
        LOGE("Couldn't create new image reader");
        return nullptr;
    }

    AImageReader_ImageListener listener{
            .context = nullptr,
            .onImageAvailable = imageCallback1,
    };

    AImageReader_setImageListener(reader, &listener);

    return reader;
}

None of the other formats are guaranteed to be supported except AIMAGE_FORMAT_JPEG, but this format doesn't seem to work with the VkSamplerYcbcrConversion because the image layout is different.

Has anyone come up against this issue before? And if so how did you resolve it?

At a high level th goal is: In C++, get the image out of the camera2 api and onto a VkImage. If anyone knows an alternative way of doing that, I'm also all ears.

Answer 1

Try to use ImageFormat.PRIVATE with USAGE_GPU_SAMPLED_IMAGE flag. This used to work fine on the mentioned Samsung devices in particular.

Please make sure to read Vulkan specification, as there are quite a few android-specific and VkSamplerYcbcrConversion requirements. I can also recommend to take a look at this great project which uses android camera2 api and vulkan.