ARCore's ArFrame_acquireCameraImage method returns green image

Question

I'm trying to get image from camera using ARCore.

I'm calling ArFrame_acquireCameraImage, which returns image with YUV_420_888 format. I also checked it using ArImage_getFormat method. It returns me 640x480 image. Then I obtain pixel stride for U plane to distinguish images with NV21 or YV12 format.

Then I combine Y, U, V arrays into single one using memcpy, encode it to Base64 (using function by J. Malinen) and print it to log.

Also I tried to perform YUV420p -> RGBA conversion using RenderScript Intrinsics Replacement Toolkit.

I have this code:

  LOGD("take frame");
  ArImage *image = nullptr;
  if (mArSession != nullptr && mArFrame != nullptr &&
      ArFrame_acquireCameraImage(mArSession, mArFrame, &image) == AR_SUCCESS) {
    const uint8_t *y;
    const uint8_t *u;
    const uint8_t *v;

    int planesCount = 0;
    ArImage_getNumberOfPlanes(mArSession, image, &planesCount);
    LOGD("%i", planesCount);

    int yLength, uLength, vLength;
    ArImage_getPlaneData(mArSession, image, 0, &y, &yLength);
    ArImage_getPlaneData(mArSession, image, 1, &u, &uLength);
    ArImage_getPlaneData(mArSession, image, 2, &v, &vLength);

    auto *yuv420 = new uint8_t[yLength + uLength + vLength];
    memcpy(yuv420, y, yLength);
    memcpy(yuv420 + yLength, u, uLength);
    memcpy(yuv420 + yLength + uLength, v, vLength);

    int width, height, stride;
    ArImage_getWidth(mArSession, image, &width);
    ArImage_getHeight(mArSession, image, &height);

    ArImage_getPlanePixelStride(mArSession, image, 1, &stride);

    //auto *argb8888 = new uint8_t[width * height * 4];

    renderscript::RenderScriptToolkit::YuvFormat format = renderscript::RenderScriptToolkit::YuvFormat::YV12;
    if(stride != 1) {
      format = renderscript::RenderScriptToolkit::YuvFormat::NV21;
    }
    LOGD("%i %i %i", width, height, format);
    
    /*renderscript::RenderScriptToolkit toolkit;
    toolkit.yuvToRgb(yuv420, argb8888, width, height, format);*/

    LOGD("%s", base64_encode(yuv420, yLength + uLength + vLength).c_str());

    // delete[](argb8888);
    delete[](yuv420);
  }
  if (image != nullptr) {
    ArImage_release(image);
  }

Full code in repo.

My phone is Xiaomi Mi A3. Also tried to run this on emulator, but it still gives me same picture.

Actual image should look like this:

However, my code prints this image (I decoded it using RAW Pixels):

Decoding parameters:

If I uncomment code for YUV420 -> ARGB conversion and print Base64 for argb8888 array, I will have this image:

Preset: RGB32, width: 640, height: 480. Base64 of this image.

Answer 1

I replaced RenderScript Intrinsics Replacement Toolkit (which have multithreading and SIMD) with code taken from TensorFlow. I see this advantages:

1. It's simpler. Here's attempt to use RSIRT:

    auto *yuv420 = new uint8_t[yLength + uLength + vLength];
    memcpy(yuv420, y, yLength);
    memcpy(yuv420 + yLength, u, uLength);
    memcpy(yuv420 + yLength + uLength, v, vLength);

    renderscript::RenderScriptToolkit::YuvFormat format = 
    renderscript::RenderScriptToolkit::YuvFormat::YV12;
    if(stride != 1) {
      format = renderscript::RenderScriptToolkit::YuvFormat::NV21;
    }

    renderscript::RenderScriptToolkit toolkit;
    toolkit.yuvToRgb(yuv420, argb8888, width, height, format);

It's line that I wrote to use TensorFlow code:

ConvertYUV420ToARGB8888(y, u, v, argb8888, width, height, yStride, uvStride, uvPixelStride);

As you see, RSIRT takes only planar image, while Tensorflow code is written to use image splitted by 3 planes, so you don't need to use memcpy. It's the reason why this decision won't hurt performance.

2. I found out that raw image is big (1.2Mb), so I shouldn't use Base64 (I think that Logcat just cut my output, so I wasn't able to see image). Now I write image to app cache and take it using adb.

Full code:

  ArImage *image = nullptr;
  if (mArSession != nullptr && mArFrame != nullptr &&
      ArFrame_acquireCameraImage(mArSession, mArFrame, &image) == AR_SUCCESS) {
    // It's image with Android YUV 420 format https://developer.android.com/reference/android/graphics/ImageFormat#YUV_420_888

    const uint8_t *y;
    const uint8_t *u;
    const uint8_t *v;

    int planesCount = 0;
    ArImage_getNumberOfPlanes(mArSession, image, &planesCount);
    LOGD("%i", planesCount);

    int yLength, uLength, vLength, yStride, uvStride, uvPixelStride;
    ArImage_getPlaneData(mArSession, image, 0, &y, &yLength);
    ArImage_getPlaneData(mArSession, image, 1, &u, &uLength);
    ArImage_getPlaneData(mArSession, image, 2, &v, &vLength);

    ArImage_getPlaneRowStride(mArSession, image, 0, &yStride);
    ArImage_getPlaneRowStride(mArSession, image, 1, &uvStride);
    ArImage_getPlanePixelStride(mArSession, image, 1, &uvPixelStride);

    int width, height;
    ArImage_getWidth(mArSession, image, &width);
    ArImage_getHeight(mArSession, image, &height);

    auto *argb8888 = new uint32_t[width * height];
    ConvertYUV420ToARGB8888(y, u, v, argb8888, width, height, yStride, uvStride, uvPixelStride);

    std::ofstream stream("/data/user/0/{your app package name}/cache/img", std::ios::out | std::ios::binary);
    for(int i = 0; i < width * height; i++)
      stream.write((char *) &argb8888[i], sizeof(uint32_t));

    stream.close();
    LOGD("%i %i", width, height);

    delete[](argb8888);
  }
  if (image != nullptr) {
    ArImage_release(image);
  }

However, I did one another thing to apply Tensorflow yuv2rgb code for my purpose. YUV2RGB inside yuv2rgb.cc have BRGA order, while Android ARGB_8888 have ARGB order. More shortly, in inline YUV2RGB method you need to change this line:

return 0xff000000 | (nR << 16) | (nG << 8) | nB;

to

return 0xff000000 | nB << 16 | nG << 8 | nR;